/llm-serving-auto-benchmark

name: llm-serving-auto-benchmark
description: Framework-independent LLM serving benchmark skill for comparing SGLang, vLLM, TensorRT-LLM, TokenSpeed, or another serving framework. Use when a user wants to find the best deployment command for one model across multiple serving frameworks under the same workload, GPU budget, and latency SLA.

LLM Serving Auto Benchmark

Overview

Use this skill to compare LLM serving frameworks such as SGLang, vLLM, TensorRT-LLM, and TokenSpeed for the same model and workload.

Use a config-driven workflow:

• keep launch-only capacity choices in each framework's `base_server_flags`
• put the search knobs in `search_space`
• run the same dataset scenarios for every framework
• generate a bounded candidate list from `search_space`, with the baseline

candidate included first

• keep failed candidates in the result file
• pick the best SLA-passing candidate after normalizing the results

For model-specific starting points, prefer the shipped configs in `configs/cookbook-llm/`. They define a framework-neutral LLM serving cookbook model set and translate each entry into framework-native SGLang, vLLM, and TensorRT-LLM, and TokenSpeed server flags. Validate those configs before a real run:

python skills/llm-serving-auto-benchmark/scripts/validate_cookbook_configs.py \
  skills/llm-serving-auto-benchmark/configs/cookbook-llm/*.yaml

If you have captured target-environment `--help` files, add `--help-dir <artifact-help-dir>`. That check only loads configs, verifies the server flag names, and renders candidate commands; it does not launch model servers.

Prefer native tooling when it gives better coverage:

• SGLang: current public cookbooks use `sglang serve`; `python -m

sglang.launch_server` remains accepted. Prefer `python -m sglang.benchmark.serving`

• vLLM: `vllm bench sweep serve` for server-parameter sweeps, otherwise

`vllm serve` plus `vllm bench serve`

• TensorRT-LLM: `trtllm-serve` for the OpenAI-compatible server plus the

TensorRT-LLM serving benchmark client or a common OpenAI-compatible benchmark client

• TokenSpeed: `tokenspeed serve` for the OpenAI-compatible server plus

`vllm bench serve --backend openai-chat` or the same OpenAI-compatible benchmark client used for the other frameworks

TensorRT-LLM has one hard scope rule in this skill: the current server is PyTorch-only; older images must select pytorch. Do not search TensorRT-LLM backend choice. If a request, config, or candidate asks for `trt`, an engine backend, or any other non-PyTorch TensorRT-LLM server backend, reject that candidate as unsupported for this skill and record the reason. This does not change the benchmark client backend; the TensorRT-LLM benchmark client still uses OpenAI-compatible modes such as `--backend openai` or `--backend openai-chat`.

Only pick a winner after each requested framework has had its main serving knobs tuned.

Framework selection is caller-controlled. If the caller explicitly supplies a framework list, benchmark only those enabled frameworks. Do not silently add TensorRT-LLM or TokenSpeed just because cookbook configs or history docs exist; record omitted frameworks as user-excluded, not unsupported.

The parameter lists in this skill are not a compatibility contract. They are version-sensitive candidate knob families. Before every real run, record the exact framework version or git commit and verify the concrete CLI flag names with `--help` in the target environment.

The default search style is framework-neutral: start from a mostly pure-TP baseline, sweep a small set of high-impact runtime knobs, and cap the first pass around 10 candidates per framework. Do not search memory fractions by default.

Source and experiment contracts

Use the [2026-09-18 source contracts](../../docs/upstream-source-contracts.md) for current profiler APIs, branch-specific defaults and KV sizing semantics. Before combining PRs or claiming kernel performance/accuracy equivalence, use [paired-validation.md](references/paired-validation.md). It distinguishes code-only A/B from deployment tuning, real from simulated acceptance, and interactivity from aggregate output throughput. Profile after timing to prove that the candidate backend actually ran.

Validation Environment

This skill is target-agnostic. It assumes any one of the following is available, and nothing more:

• a local GPU host with Docker/Podman and the target framework images pulled;
• a remote GPU host reached via `ssh <host>` with the framework images already

running in a container there;

• a CI runner that can exec into a pre-built image for each framework.

Do not assume a specific operator host name inside this skill's own workflow. The concrete SSH wiring, container names, workspace paths, and HF token plumbing for a given box live in operator-side per-host skills; this skill only requires that the caller can reach a shell inside a container with the requested framework installed.

Reference files are optional and version-sensitive. Treat historical flag notes as evidence from one image, not as a compatibility guarantee for the next run.

Historical smoke validation on 2026-05-01 (H100) and 2026-06-27 (B200) exercised small serving/search workloads. The B200 image provided only SGLang; missing vLLM/TRT-LLM/TokenSpeed CLIs were environment gaps. Those runs are not current cross-framework speed or accuracy results. The September source refresh ran offline tooling, not a new GPU matrix.

Skill Scope

This skill is a playbook plus a config+validator toolchain, not a turn-key orchestrator. The operator still launches servers, drives workloads, and writes one normalized JSONL row per candidate.

The `scripts/` directory contains exactly two tools:

• `validate_cookbook_configs.py`: load cookbook YAML, render bounded candidate

server commands, and check flag names against captured `--help` snapshots without launching servers.

• `compare_benchmark_results.py`: turn no