/deployment

name: deployment
description: Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says "deploy model", "serve model", "start vLLM server", "launch SGLang", "TRT-LLM deploy", "AutoDeploy", "benchmark throughput", "serve checkpoint", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).
license: Apache-2.0

Deployment Skill

Serve a model checkpoint as an OpenAI-compatible inference endpoint. Supports vLLM, SGLang, and TRT-LLM (including AutoDeploy).

Quick Start

Prefer `$SKILL_DIR/scripts/deploy.sh` for standard local deployments — it handles quant detection, health checks, and server lifecycle. Use the raw framework commands in Step 4 when you need flags the script doesn't support, or for remote deployment.

# Start vLLM server with a ModelOpt checkpoint
"$SKILL_DIR/scripts/deploy.sh" start --model ./qwen3-0.6b-fp8

# Start with SGLang and tensor parallelism
"$SKILL_DIR/scripts/deploy.sh" start --model ./llama-70b-nvfp4 --framework sglang --tp 4

# Start from HuggingFace hub
"$SKILL_DIR/scripts/deploy.sh" start --model nvidia/Llama-3.1-8B-Instruct-FP8

# Test the API
"$SKILL_DIR/scripts/deploy.sh" test

# Check status
"$SKILL_DIR/scripts/deploy.sh" status

# Stop
"$SKILL_DIR/scripts/deploy.sh" stop

The script handles: GPU detection, quantization flag auto-detection (FP8 vs FP4), server lifecycle (start/stop/restart/status), health check polling, and API testing.

Decision Flow

0. Check workspace (multi-user / Slack bot)

If `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md`. Before creating a new workspace, check the current session for existing model workspaces — especially if deploying a checkpoint from a prior PTQ run:

ls "$MODELOPT_WORKSPACE_ROOT/<session_id>/" 2>/dev/null

If the user says "deploy the model I just quantized" or references a previous PTQ, find the matching workspace and `cd` into it. The checkpoint should be in that workspace's output directory.

1. Identify the checkpoint

Determine what the user wants to deploy:

• **Local quantized checkpoint** (from ptq skill or manual export): look for `hf_quant_config.json` in the directory. If coming from a prior PTQ run in the same workspace, check common output locations: `output/`, `outputs/`, `exported_model/`, or the `--export_path` used in the PTQ command.
• **HuggingFace model hub** (e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`): use directly
• **Unquantized model**: deploy as-is (BF16) or suggest quantizing first with the ptq skill

> **Note:** This skill expects HF-format checkpoints (from PTQ with `--export_fmt hf`). TRT-LLM format checkpoints should be deployed directly with TRT-LLM — see `references/trtllm.md`.

Check the quantization format if applicable:

cat <checkpoint_path>/hf_quant_config.json 2>/dev/null || echo "No hf_quant_config.json"

If not found, also check `config.json` for a `quantization_config` section with `quant_method: "modelopt"`. If neither exists, the checkpoint is unquantized.

2. Choose the framework

If the user hasn't specified a framework, recommend based on this priority:

| Situation | Recommended | Why | |-----------|-------------|-----| | General use | **vLLM** | Widest ecosystem, easy setup, OpenAI-compatible | | Best SGLang model support | **SGLang** | Strong DeepSeek/Llama 4 support | | Maximum optimization | **TRT-LLM** | Best throughput via engine compilation | | Mixed-precision / AutoQuant | **TRT-LLM AutoDeploy** | Only option for AutoQuant checkpoints |

Check the support matrix in `references/support-matrix.md` to confirm the model + format + framework combination is supported.

3. Check the environment

Use the common skill's `environment-setup.md` for GPU detection, local vs remote, and SLURM/Docker/bare metal detection. After completing it you should know: GPU model/count, local or remote, and execution environment.

Then check the **deployment framework** is installed:

python -c "import vllm; print(f'vLLM {vllm.__version__}')" 2>/dev/null || echo "vLLM not installed"
python -c "import sglang; print(f'SGLang {sglang.__version__}')" 2>/dev/null || echo "SGLang not installed"
python -c "import tensorrt_llm; print(f'TRT-LLM {tensorrt_llm.__version__}')" 2>/dev/null || echo "TRT-LLM not installed"

If not installed, consult `references/setup.md`.

**GPU memory estimate** (to determine tensor parallelism):

• BF16: `params × 2 bytes` (8B ≈ 16 GB)
• FP8: `params × 1 byte` (8B ≈ 8 GB)
• FP4: `params × 0.5 bytes` (8B ≈ 4 GB)
• Add ~2-4 GB for KV cache and framework overhead

If the model exceeds single GPU memory, use tensor parallelism (`-tp <num_gpus>`).

4. Deploy

Read the framework-specific reference for detailed instructions:

| Framework | Reference file | |-----------|---------------| | vLLM | `references/vllm.md` | | SGLang | `references/sglang.md` | | TRT-LLM | `references/trtllm.md` |

**Quick-start commands** (for common cases):

vLLM

# Serve as OpenAI-compatible endpoint
python -m vllm.entrypoints.openai.api_server \
    --model <checkpoint_path> \
    --quantization modelopt \
    --tensor-parallel-size <num_gpus> \
    --host 0.0.0.0 --port 8000

For NVFP4 checkpoints, use `--quantization modelopt_fp4`.

> **NVFP4 on Blackwell B300/GB300 (sm_103) needs a CUDA-13 image.** From v0.20.0 > on, release tags are CUDA-13 unsuffixed (e.g. `vllm/vllm-openai:v0.26.0`) with > `-cu129` the CUDA-12 opt-out; v0.19.x and earlier were the other way round > (`-cu130` = CUDA 13), and `-cu130` no longer exists after v0.20.0. Don't trust > the tag name — select a tag reporting `CUDA_VERSION` >= 13 in the config blob > of your platform's child manifest (arm64 Grace/GB300, amd64 x86); > `TORCH_CUDA_ARCH_LIST` differs between the two. A cu12 build has **no sm_103 >