/agentsop-vllm

name: agentsop-vllm
description: Decision SOP for serving LLMs with vLLM. Covers PagedAttention mental model, quantization/parallelism/batching tradeoffs, OOM triage, and when NOT to use vLLM. Activates when a coder-agent is choosing or tuning an inference engine, debugging vLLM throughput/latency/OOM, or comparing vLLM against TGI/SGLang/TensorRT-LLM/llama.cpp.
domain: high-throughput LLM inference serving
version: 1.0
sources:
  - https://arxiv.org/abs/2309.06180
  - https://docs.vllm.ai/en/stable/
  - https://github.com/vllm-project/vllm
  - https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance

vLLM Serving SOP

1. 何时激活 (When to activate)

Activate this skill when any of the following hold:

• The user wants to **serve an LLM in production** (multi-user, concurrent requests, throughput-oriented) and has GPU infrastructure.
• The user is **comparing inference engines** (vLLM vs TGI vs SGLang vs TensorRT-LLM vs llama.cpp/Ollama).
• The user reports a **vLLM operational issue**: CUDA OOM, low throughput, high TTFT, request preemption, multi-GPU setup, quantization choice.
• The user is **sizing hardware** for an open-weights model (Llama / Qwen / Mixtral / DeepSeek-V3) and asking about tensor/pipeline parallelism.
• The user mentions PagedAttention, prefix caching, continuous batching, chunked prefill, or speculative decoding.

**Do NOT activate** for: training/fine-tuning (use accelerate/deepspeed/trl), CPU-only edge inference (use llama.cpp/Ollama), Apple Silicon production (vLLM Metal/MPS is experimental, not production-ready as of 2026) [aimadetools.com 2026], API-only consumption of hosted models (just call the OpenAI/Anthropic SDK).

---

2. 核心心智模型 (Core Mental Model)

2.1 The OS abstraction: KV cache as virtual memory

vLLM's defining insight (Kwon et al., SOSP 2023) is that **LLM serving's bottleneck was not compute — it was KV-cache memory fragmentation**. Pre-vLLM systems pre-allocated a contiguous KV-cache slot per request, sized for the maximum possible output length; in early 2023, inference engines used only **20–40% of available GPU memory** because of internal+external fragmentation [arxiv.org/abs/2309.06180; zilliz.com/learn].

PagedAttention applies classic OS paging to KV cache:

• **Block** = fixed-size chunk of KV cache (default 16 tokens; ~12.8 KB for a 13B model) [medium.com/@mandeep0405].
• **Logical blocks** per request → **block table** → **physical blocks** in GPU memory (analogous to virtual→physical page table).
• Blocks need not be contiguous. The attention kernel reads scattered physical blocks via the block table and presents them as a logical contiguous sequence.
• **Copy-on-write** + **prefix sharing**: multiple requests that share a prefix (e.g. a system prompt, few-shot examples) share KV blocks; a write triggers a per-request copy [arxiv.org/abs/2309.06180].

**Result**: near-zero memory waste → larger batch sizes → **2–4× throughput** vs FasterTransformer/Orca at equal latency [arxiv.org/abs/2309.06180]; **14–24×** vs vanilla HuggingFace Transformers [yottalabs.ai 2026].

2.2 The scheduler: iteration-level (continuous) batching

vLLM inherits Orca's iteration-level scheduling (OSDI 2022, **36.9× over FasterTransformer** [medium.com/byte-sized-ai]). Instead of waiting for a static batch to finish, the scheduler reassigns batch slots **every decode step**: a request that finishes early frees its slot to a waiting request. Static batching is dead; continuous batching is table stakes.

2.3 Prefill vs decode are different beasts

• **Prefill** (processing the prompt): compute-bound, high SM utilization, scales with input length.
• **Decode** (generating tokens one-at-a-time): memory-bandwidth-bound, low SM utilization.
• A long prefill blocks all decodes on the GPU → **head-of-line blocking** → high inter-token latency (ITL) for already-streaming requests.
• **Chunked prefill** breaks long prefills into pieces interleaved with decode steps [docs.vllm.ai/en/stable/configuration/optimization/].

**Takeaway**: when tuning, separate TTFT (time-to-first-token, gated by prefill+queue) from ITL (gated by decode bandwidth and batch interference).

2.4 Three throughput levers, in order of impact

Per Red Hat's tuning hierarchy [developers.redhat.com 2026]: 1. **Right-size the model** (smallest adequate). 2. **Scale hardware** (more replicas; better-bandwidth GPUs). 3. **Quantize** (FP8 weights + KV cache). 4. **Speculative decoding** (model-based: EAGLE-3 / MTP). 5. **Refine parallelism** (TP degree, replicas vs higher TP).

---

3. SOP 工作流 (SOP Workflow)

[Step 0] Confirm vLLM is the right tool
   ├─ Production, GPU-backed, concurrent users? → continue
   └─ Else → see §7 (ecosystem) and stop

[Step 1] Pick the model + precision
   ├─ Model fits in single-GPU VRAM at BF16?         → keep BF16, TP=1
   ├─ Need 50% VRAM cut, ~zero quality loss?         → FP8 (Hopper/Ada+) [arxiv 2411.02355]
   ├─ Need 4× VRAM cut, tolerate ~1.6pt avg drop?    → AWQ-4 or GPTQ-4
   └─ Reasoning-heavy / coding workload?             → favor FP8 > AWQ; verify on eval set

[Step 2] Choose parallelism
   ├─ Fits 1 GPU                                     → TP=1, PP=1
   ├─ Fits 1 node, NVLink present                    → TP=#GPUs/node
   ├─ Fits 1 node, only PCIe (e.g. L40S)             → PP within node (TP-only over PCIe collapses)
   ├─ Multi-node                                     → TP=GPUs/node, PP=#nodes
   └─ MoE model (Mixtral, DSv3)                      → DP attention + EP/TP for MoE layers

[Step 3] Set memory/batch envelope
   ├─ --gpu-memory-utilization 0.90 (default; 0.85 if sharing GPU)
   ├─ --max-model-len = (longest realistic prompt + output) — NOT model max!
   ├─ --max-num-seqs (start 256; lower if preemption logs appear)
   └─ --max-num-batched-tokens (raise for TTFT; lower for ITL)

[Step 4] Turn on the free wins
   ├─ enable_prefix_caching=True            → if any system-prompt/few-shot reuse
   ├─ en