/llm-serving-capacity-planner

name: llm-serving-capacity-planner
description: "Parse SGLang/vLLM startup logs to explain GPU memory use and request capacity. Use for KV cache budget, mem-fraction-static comparisons, OOM triage, and max-concurrency estimates."

LLM Serving Capacity Planner

Overview

Use this when a serving log has enough memory lines to explain where GPU HBM went. The analyzer reads SGLang/vLLM startup logs, extracts weight load, KV pool, CUDA graph, framework overhead, and token-capacity lines, then estimates concurrent requests for common token lengths.

For DeepSeek-V4.1, read the [KV layout and page-allocation contract](../llm-torch-profiler-analysis/references/fuse-overlap-catalog.md#paged-kv-storage-contract) before using 528 B/288 B in an estimate. These are per-stored-token payload and scale sizes, not allocation after page padding or total model KV bytes. Verify actual pool layout, compression ratios and sharing from the pinned source and logs.

Inputs

Before running analysis, collect or verify these inputs:

| Item | Why it matters | How to obtain | Default if user skips | |---|---|---|---| | Log file path | Primary input; all memory data comes from here | Ask user for the serving startup log | — (required) | | GPU type | Determines total HBM for decomposition validation | Ask user or infer from log | Auto-detected from log if possible | | nvidia-smi output | Provides per-rank actual memory for cross-validation | Capture with `nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader > smi.txt` | — (optional, but recommended) | | Model config.json | Enables theoretical KV cache byte calculation and replication factor analysis | Ask user for the model's config.json path | — (optional, log data used instead) | | Request token length | Determines concurrency estimate denominator | Ask user | 4096, 6144, 8192 |

Workflow

Step 1: Collect the serving log

The user should provide the startup log from an SGLang or vLLM serving instance. Key log lines that the analyzer needs:

• `Load weight begin. avail mem=XX GB`
• `server_args={'model_path': '...', 'tp_size': 8, ...}` (resolved parameters)
• `Load weight end. ... avail mem=X GB, mem usage=X GB.` (weight footprint)
• `DSV4 memory calculation: ... bytes_per_full_token=X, available_bytes=X GB, ... full_token=N`
• `[label] KV Cache is allocated. dtype: ..., #tokens: N, K size: X GB, V size: Y GB` (or `KV size`)
• `Mamba Cache is allocated. ... conv_state size: X GB, ssm_state size: Y GB`
• `Memory pool end. avail mem=XX GB`
• `Capture target|draft decode|verify|prefill CUDA graph end. elapsed=X s, mem usage=X GB, avail mem=X GB.`
• `Post-capture KV sizing: KV cache allocated. ... KV size: X GB, avail mem=X GB` (opt-in)
• `max_total_num_tokens=XX, ... chunked_prefill_size=-1|N, ... available_gpu_mem=XX GB`

Legacy `ServerArgs(...)`, `Memory profiling:` and `Capture cuda graph end.` remain accepted for older/private branch logs; they are not current main emitters.

Current vLLM V1 logs instead expose:

• `Initial free memory: ...; Requested memory: ...` (DEBUG only)
• `Model loading took ... GiB`
• `Available KV cache memory: ... GiB`
• `Graph capturing finished ... took ... GiB`
• `<device> KV cache size: ... tokens, Maximum concurrency for ... tokens per request: ...x`
• `Free memory on device (F/T GiB) ... Current kv cache memory in use is K GiB` (INFO)
• `CUDA graph pool memory: A GiB (actual), E GiB (estimated), ...`
• `Initial free memory X GiB, reserved Y GiB memory for KV Cache ...` (explicit KV bytes)
• `Maximum concurrency for ... tokens per request: ...x`

If the log is from a running instance, capture it by redirecting stdout/stderr to a file at launch time.

Step 2: Optionally capture nvidia-smi data

For per-rank memory comparison:

docker exec <container> nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader > smi.txt

Step 3: Run the analyzer

python3 skills/llm-serving-capacity-planner/scripts/capacity_analyzer.py \
  --log-file /path/to/sglang.log \
  --nvidia-smi-file /path/to/smi.txt \
  --gpu h200 \
  --config-json /path/to/config.json

For JSON output (automation):

python3 skills/llm-serving-capacity-planner/scripts/capacity_analyzer.py \
  --log-file /path/to/sglang.log \
  --format json

Step 4: Review and interpret results

The analyzer prints:

1. **Memory breakdown table**: each category (weights, KV pool, CUDA graph, framework, other) with GiB, MiB, percentage, and derivation 2. **Per-rank comparison**: nvidia-smi data across all TP ranks 3. **KV pool detail**: pool configuration, KV dtype, replication factor, per-token byte calculation 4. **Concurrency estimate**: max concurrent requests for different token lengths 5. **Tuning notes**: configuration changes that may increase capacity

For vLLM, the report also includes a `vLLM Startup Evidence` section. Values that are not present in the log stay unknown; do not translate them into SGLang-only fields.

When To Use It

• After launching an LLM serving instance, to understand how GPU memory is distributed
• When comparing different `--mem-fraction-static` values and their impact on KV pool capacity
• When planning deployment capacity: how many concurrent requests can a given GPU configuration support
• When investigating OOM issues: identifying which memory category is consuming the most
• When evaluating whether fp8 KV cache or EP can improve concurrency

Key Concepts

mem-fraction-static

The inspected SGLang source reserves runtime headroom from **pre-model-load free memory**: `headroom = pre_model_load_memory * (1 - mem_fraction_static)`. The KV budget starts from currently free memory minus this headroom and other model-specific reservations. It is not `post_weight_free * fraction`. Post-capture sizing may update the pool again. Use final logged capacity and rank-local evidence; see [source contracts](../../docs/upstream-source-contracts.md).

The def