/sglang-prod-incident-triage

name: sglang-prod-incident-triage
description: Replay-first debug flow for SGLang serving problems. Use when a live or recent server shows health-check failures, latency or throughput regressions, queue growth, timeouts, distributed stalls, crash dumps, wrong outputs after deploys, or PD/EP/HiCache issues, and the job is to turn the problem into a replay plus the right next debug tool.

SGLang Serving Debug

Use the maintained [source contracts](../../docs/upstream-source-contracts.md) when applying historical evidence to a current branch. The review/history corpora retain their own capture dates; they do not certify today's dispatch or numerical defaults. For kernel replacements, verify that the real model executes the candidate before treating end-to-end tests as coverage.

Overview

Use this skill to turn a live serving problem into a debug path you can replay.

Use one loop:

• collect a baseline bundle
• save the failing request or crash dump
• replay on a clean target
• only then switch tools

Do not start with profiling.

As of SGLang `v0.5.21` (2026-09-29), two extra live-server signals matter:

• `--enable-session-radix-cache` plus `/close_session`: agent or RL rollouts

that forget to close sessions can look like unexplained KV growth

• the opt-in Rust ingress (`SGLANG_RUST_SERVER=1`) has no API-key boundary; if

the incident is auth-related, confirm which server binary is actually serving before blaming application tokens

This skill should work with more focused skills instead of re-implementing them:

• `debug-cuda-crash` when replay plus coredump points to a CUDA crash path
• `debug-distributed-hang` when the problem is clearly a TP/PP/DP/EP hang
• `llm-torch-profiler-analysis` when the issue is already narrowed to a

compute-side path

Three examples are included:

• TTFT spike with low queue time
• replay-first CUDA crash flow
• request-shaped distributed hang flow

Output Contract

Return:

• problem class
• what was checked
• strongest signal so far
• current best guess
• what was ruled out
• next step
• production risk

When To Use It

• `/health` or `/health_generate` is unhealthy
• latency or throughput regressed under serving load
• queue size grows while health still looks green
• one request class times out or hangs
• the server crashes only after some requests
• outputs changed after a deploy, topology change, or weight switch
• one older commit is known-good and a newer commit is known-bad

Workflow

1. Collect a baseline bundle

Confirm the ingress first: Python `/server_info.frontend` is `python`. Rust ingress lacks `/v1/loads`, `/metrics`, `/configure_logging`, `/hicache/*`, `/set_trace_level`, `/start_profile` and `/ready`; 404 there is an ingress limitation. Rust health stays 503 until startup warmup succeeds.

If a live server is reachable, collect a read-only bundle before anything more intrusive:

python3 scripts/incident_artifact_tool.py collect-bundle \
  --base-url http://127.0.0.1:30000 \
  --outdir /tmp/incident_bundle

python3 scripts/incident_artifact_tool.py summarize-bundle \
  /tmp/incident_bundle

If the server is protected:

python3 scripts/incident_artifact_tool.py collect-bundle \
  --base-url http://127.0.0.1:30000 \
  --token "$SGLANG_BEARER_TOKEN" \
  --outdir /tmp/incident_bundle

The bundle script collects:

• `/ready`
• `/health`
• `/health_generate`
• `/model_info`
• `/server_info`
• `/v1/loads?include=all`
• `/v1/loads?include=core,queues,disagg,spec`
• `/metrics`
• `/hicache/storage-backend` on a best-effort basis

If stale or corrupt HiCache contents are the working diagnosis, capture the bundle and relevant request dumps before calling `POST /hicache/storage-backend/clear`. That endpoint clears the attached backend's contents without detaching it; treat it as destructive and use it only when losing the cached contents is acceptable. Attach, detach, and clear details are in [references/endpoints-and-signals.md](references/endpoints-and-signals.md).

Use the summary for a quick read on:

• health vs. active health state
• topology and runtime flags
• point-in-time queue and token usage
• TTFT / E2E / queue-time heuristics from Prometheus metrics

If the summary says the bundle was captured while the server was idle, recollect it during traffic or move quickly to dump plus replay.

If no live server is reachable, start from the best dump or log already available:

• crash dump
• request dump
• logs
• CUDA coredump
• OTel trace
• torch profile

2. Save the failing request

Read [references/decision-tree.md](references/decision-tree.md) only if the problem class is still unclear:

• server down or unhealthy
• latency or throughput regression
• wrong output or behavior regression
• intermittent timeout or hang

Then preserve the request payload that actually triggers the problem:

• crash path: use `--crash-dump-folder`
• non-crash path: enable request dump or save the exact trigger request

Do not jump straight from a live symptom to low-level debugging without first saving something you can replay.

3. Replay on a clean target

Read [references/endpoints-and-signals.md](references/endpoints-and-signals.md) when you need help reading the baseline bundle or the replay target.

Read [references/replay-trace-profile.md](references/replay-trace-profile.md) when you need the replay, trace, profile, or bisect paths.

Standard order:

1. collect baseline bundle 2. capture request dump or crash dump 3. restart a clean debug target if needed 4. replay the same issue 5. collect replay-time logs and dumps

4. Only go deeper after replay

Replay

Use replay when:

• a crash dump exists
• a request dump exists
• the problem depends on request shape or workload mix

If a crash dump exists, summarize it first:

python3 scripts/incident_artifact_tool.py summarize-dump \
  --input-file /path/to/crash_dump.pkl

Then replay:

python3 /path/to/sglang/scripts/playg