/jetson-llm-benchmark

name: jetson-llm-benchmark
description: Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.
version: 0.0.2
license: "Apache-2.0"
metadata:
  author: "Jetson Team"
  tags: [jetson, llm, benchmark]
  languages: [bash]
  data-classification: public

Jetson LLM Benchmark

Reproducible Jetson benchmarks with **structured JSON output** so an agent can compare runs. Encodes the workflow from the [Jetson AI Lab GenAI Benchmarking tutorial](https://www.jetson-ai-lab.com/tutorials/genai-benchmarking/).

Purpose

Measure deployed LLM latency and throughput on a Jetson target using the correct runtime-specific benchmark wrapper. Use the JSON output to compare models, runtime flags, power modes, and before/after tuning changes.

Prerequisites

• Run on the Jetson device that hosts the model runtime.
• For vLLM, start the OpenAI-compatible vLLM server first and know the served

model ID.

• For Ollama, ensure the Ollama daemon is reachable at `--endpoint` and the

named model is already pulled.

• For llama.cpp/GGUF, provide a readable `.gguf` model path on the host.
• Put the device in the intended power mode before measuring. MAXN is preferred

for comparable performance numbers.

Available Scripts

| Script | Purpose | Arguments | |--------|---------|-----------| | `scripts/bench_vllm.sh` | Runs `vllm bench serve` against a running OpenAI-compatible vLLM server. | `--model`, `--endpoint`, `--concurrency`, `--input-len`, `--output-len`, `--num-prompts`, `--no-warmup`, `--container`, `--native`. | | `scripts/bench_llama_cpp.sh` | Runs `llama-bench` for a local GGUF model through the Jetson-appropriate NVIDIA-AI-IOT llama.cpp container. | `--model`, `--n-prompt`, `--n-gen`, `--n-gpu-layers`, `--threads`, `--container`. | | `scripts/bench_ollama.sh` | Benchmarks a local or containerized Ollama daemon through the `/api/generate` REST API. | `--model`, `--endpoint`, `--num-prompts`, `--input-len`, `--output-len`, `--no-warmup`. |

If your agent runtime supports `run_script`, invoke the selected wrapper directly with the user-provided model identifier or local model path, then summarize the returned JSON. Otherwise run the wrapper with `bash {baseDir}/scripts/<wrapper-name> ...`.

Instructions

Always use the matching wrapper script for the runtime — do **not** call the underlying `vllm bench serve`, `llama-bench`, or `curl` against `/api/generate` by hand:

• vLLM → `scripts/bench_vllm.sh` (required for the vLLM path)
• llama.cpp / GGUF → `scripts/bench_llama_cpp.sh` (required for the GGUF path)
• Ollama → `scripts/bench_ollama.sh` (required for the Ollama path)

These wrappers handle warmup, the NVIDIA-AI-IOT container selection, and JSON emission. Calling the underlying tool directly will not satisfy the output contract below.

For "how do I benchmark/measure" questions, first run the matching wrapper with `--help` to verify the exact options, then answer with the wrapper command. Do not run a full benchmark unless the user asks you to execute it or the required server/model path is already confirmed.

Expected Workflow

Pick exactly one wrapper based on the runtime the user named, and invoke that wrapper with `--help` before composing the answer. Do not merely mention the script name. If the runtime does not execute scripts relative to the skill directory, use `{baseDir}/scripts/<wrapper-name>`.

• Existing vLLM OpenAI-compatible server at `localhost:8000`:

`{baseDir}/scripts/bench_vllm.sh --help`, then show a command using `--concurrency 1,8` and the served model ID.

• llama.cpp / GGUF / `llama-server`: `{baseDir}/scripts/bench_llama_cpp.sh

--help`, then show a command for the GGUF model path and report that prompt/generation speed maps to TTFT, ITL/TPOT, and throughput.

• Ollama: `{baseDir}/scripts/bench_ollama.sh --help`, then show a command with

`--model <ollama-tag>`. Do not use vLLM or llama.cpp wrappers for Ollama.

When to use

• "Benchmark / measure / compare X on this Jetson."
• After `jetson-llm-serve` to actually quantify the deployment.
• Before/after applying flags from `jetson-inference-mem-tune` to confirm the change helped.

Three paths — pick by runtime

A. vLLM (preferred for parity with how things are served)

Server must already be running (use `jetson-llm-serve`). Run **`bench_vllm.sh`**:

scripts/bench_vllm.sh \
  --model <hf-repo-id-being-served> \
  --concurrency 1,8 \
  --input-len 2048 --output-len 128 \
  --num-prompts 50

Uses the Jetson-appropriate benchmark client path: upstream vLLM 0.20+ container `vllm/vllm-openai:latest` on Thor and Orin JetPack 7.2 / L4T r39+, or the NVIDIA-AI-IOT vLLM benchmark container `ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin` on older Orin. Pass `--native` only when host-native vLLM is already installed and validated. It runs against `http://localhost:8000/v1`. **Always do a warmup pass first** (~10 prompts, discarded) before the measured run — Jetson has cold caches and JIT'd kernels.

B. Ollama (for models served by a running Ollama daemon)

No benchmark container needed. Uses Ollama's `/api/generate` REST API directly — timing data (TTFT, ITL, throughput) comes from the response JSON, so no `--verbose` parsing is required.

**Prerequisite:** the Ollama daemon must be reachable at `--endpoint` (default `http://localhost:11434`). This works whether Ollama is installed natively or running in a container that exposes that port. If the daemon is not running, the script will tell you whether Ollama is installed but stopped (`ollama serve` to fix) or not installed at all (install instructions printed). Run **`bench_ollama.sh`** (do not roll your own `curl` against `/api/generate`):

scripts/bench_ollama.sh \
  --model <ollama-model-name> \
  --num-prompts 20 \
  --input-len 512 --output-len 128

Runs sequential single-stream requests (concurrency=1). Ollama is a single-stream runtime by design, so multi-concurrency