/jetson-llm-serve

name: jetson-llm-serve
description: Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin.
version: 0.0.1
license: "Apache-2.0"
metadata:
  author: "Jetson Team"
  tags: [jetson, llm, serving]
  languages: [markdown]
  data-classification: public

Jetson LLM Serve

Encodes the [Jetson AI Lab GenAI tutorial](https://www.jetson-ai-lab.com/tutorials/genai-on-jetson-llms-vlms/): on Orin JetPack 7.2 / L4T r39+, use upstream vLLM 0.20+ (`vllm/vllm-openai:latest`); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (`nvcr.io/nvidia/sglang:26.01-py3`, SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials/cache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs.

Purpose

Provide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.

When to use

• "Run / serve / host this model on a Jetson."
• "Start a vLLM server I can hit from Open WebUI / my app."
• After `jetson-inference-mem-tune` produced launch flags and the user wants to actually start the server.

For recipe-only questions, answer from this document without starting containers. Run live pre-flight checks only when the user asks you to check this device or execute the deployment.

Prerequisites

• Run on the Jetson host or a shell with Docker access to the Jetson GPU runtime.
• Know the target Jetson generation (`thor` or `orin`) and the model identifier or local checkpoint path.
• Use `HF_TOKEN` only when the model is gated/private; public models should omit the token environment variable.
• Use `jetson-inference-mem-tune` first when memory headroom or launch flags are uncertain.

Instructions

For recipe questions, provide a complete launch recipe instead of trying to call `jetson-llm-serve` as a tool. A complete answer includes:

• The Jetson-appropriate runtime path: upstream vLLM 0.20+ (`vllm/vllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io/nvidia/sglang:26.01-py3`, SGLang 0.5.5.post2) on Thor, NVIDIA-AI-IOT vLLM container on older Orin, or upstream vLLM 0.20+ on Orin JetPack 7.2 / L4T r39+.
• The model checkpoint / Hugging Face repo the user named.
• A `docker run` + server command sketch with `--host 0.0.0.0 --port 8000`.
• The OpenAI-compatible endpoint: `http://<jetson-ip>:8000/v1`.
• A verification step such as `curl http://localhost:8000/v1/models`.

For VLM questions, explicitly say the VLM uses the same vLLM serving flow as an LLM with a different vision-language checkpoint. Do not omit `vLLM` or the Jetson container when answering VLM prompts.

Step 1 — Pick the runtime path (per Jetson family)

Use upstream vLLM 0.20+ on Thor (`vllm/vllm-openai:latest`, or a validated native vLLM 0.20+ install). On Orin JetPack 7.2 / L4T r39+, use upstream vLLM 0.20+ (`vllm/vllm-openai:latest`). On older Orin releases, use the **NVIDIA-AI-IOT prebuilt vLLM image** ([packages](https://github.com/orgs/NVIDIA-AI-IOT/packages)) because it ships the correct CUDA / cuDNN / TensorRT stack for that JetPack. Use NVIDIA SGLang 26.01 (`nvcr.io/nvidia/sglang:26.01-py3`, SGLang 0.5.5.post2) on Thor when the user asks for SGLang, RAG, tool-use, or programmable serving; do not recommend native upstream SGLang on Orin unless a JetPack-matched release explicitly supports it.

| Jetson family | Runtime path | |-----------------------------|---------------------------------------------------| | Thor (T5000, T4000) | upstream vLLM 0.20+ (`vllm/vllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io/nvidia/sglang:26.01-py3`, SGLang 0.5.5.post2) | | AGX Orin / Orin NX / Nano | Orin JetPack 7.2 / L4T r39+: upstream vLLM 0.20+ (`vllm/vllm-openai:latest`); older Orin: `ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin` |

To detect the silicon era for image tags:

1. Source the detector so exports survive in your shell:

   . skills/jetson-diagnostic/scripts/detect_jetson.sh

2. Check `JETSON_GENERATION` (`thor` or `orin`) and choose the matching runtime path from the table above. 3. Use `JETSON_PRODUCT_LINE` for a finer bucket such as `thor-agx` or `orin-nano`; `JETSON_SKU` remains the legacy identifier.

Do not use `bash skills/jetson-diagnostic/scripts/detect_jetson.sh` when you need exported variables in the caller; running with `bash` uses a subshell.

Step 2 — Set MAXN power mode

sudo nvpmodel -m 0 && sudo jetson_clocks

Skip this only if the user explicitly asks for a power-constrained run; otherwise benchmark and serving numbers will be inconsistent.

Step 3 — Run the server

On Thor with vLLM, use upstream vLLM 0.20+ (`vllm/vllm-openai:latest`) or a validated native vLLM 0.20+ install:

docker run --rm -it --runtime nvidia --network host --ipc host --name vllm \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e HF_TOKEN="$HF_TOKEN" \
  vllm/vllm-openai:latest \
  vllm serve <hf-repo-id> \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.75 \
    --tensor-parallel-size 1

On Orin JetPack 7.2 / L4T r39+, use upstream vLLM 0.20+ (`vllm/vllm-openai:latest`). On older Orin releases, use the NVIDIA-AI-IOT container:

docker run --rm -it --runtime nvidia --network host --name vllm \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e HF_TOKEN="$HF_TOKEN" \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin \
  vllm serve <hf-repo-id> \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.85 \
    --tensor-parallel-size 1

`HF_TOKEN` is required only for gated/private Hugging Face models; omit the `-e HF_TOKEN="$HF_TOKEN"` line for public models that do not need Hub authentication. Passing