/omh-inference-serving

name: "omh-inference-serving"
description: "[omh] Self-hosted LLM serving on GPUs: choose the serving engine and quantization from decision tables, prepare deployment as an idempotent runbook with observed-only verification, and measure the endpoint with the standard TTFT/TPOT/goodput protocol. Use when the user says: inference-serving, inference serving, serve this model, serve the model, model serving, serving endpoint, vllm, llama.cpp."
metadata:
  hermes:
    tags: [workflow, oh-my-hermes, operations]
    category: operations
    phase: inference-serving
    role: operator
    quality_tier: observed-command-gated

Inference Serving

This is a Hermes-native `inference-serving` workflow skill.

Why This Exists

`inference-serving` exists so serving an LLM runs as one decided, gated, measured process instead of scattered flag folklore: the engine choice is a table, the deployment is an idempotent runbook whose only completion evidence is the observed verification, and the benchmark speaks the standard metric vocabulary.

Do Not Use When

• A new model generation needs recognition, calibration, routing, and pricing onboarding; use `model-optimization`.
• The user wants their own machine's model routing or providers configured; use `model-setup`.
• The question is whether a coding runtime/executor can run at all; use `executor-runtime-readiness`.
• The goal is application or system performance rather than the serving endpoint itself; use `ultraperf`.

Examples

Good example:

• Prompt: Serve Qwen on our two A100s for the team and tell me if prefix caching is worth turning on.
• Expected behavior: Engine verdict (vLLM, TP as a power of two), quantization check, the k8s or docker runbook with its gates and verification, then the prefix-cache A/B protocol with hit-rate assumptions recorded - numbers only from observed runs.
• Why: Serving plus a measured tuning question is exactly the decide-deploy-measure process this workflow owns.

Bad example:

• Prompt: Just tell me the endpoint is fast enough, we already know it works.
• Expected behavior: Refuse the unmeasured claim; run the benchmark protocol against the stated SLO or report the capacity question as unanswered.
• Why: A fast-enough claim without a load shape and observed results is the folklore this skill replaces.

Completion Checklist

• The engine/quantization verdict names the situation-table row it came from and the rejected options.
• Every runbook step's status is prepared or observed, never assumed, and the port invariant was honored.
• Benchmark numbers carry metrics, load shape, dataset, SLO, and saved metadata, or are not reported.
• Anything the workflow started for measurement was stopped, and credentials never appear in artifacts.

Recovery Notes

• If the hardware truth is unknown, probe it first (GPU inventory, VRAM) instead of assuming the engine.
• If deployment verification fails, walk the failure ladder (toolkit, shared memory, permissions, token) before editing manifests.
• If a benchmark misses the verify targets, go to the symptom->flag table and re-measure one change at a time.

Workflow Lane

• Current lane: **Research and company ops** (`product-docs`, `source-finder`, `web-research`, `research`, `model-optimization`, `inference-serving`, `model-finetuning`, `research-brief`, `+20 more`) - research, signals, ops, and briefings.
• If intent belongs to another lane, hand back to `oh-my-hermes` or name the adjacent workflow.
• Shared product, routing, compatibility, and evidence rules: `omh-routing/references/skill-common-rail.md`.

Use When

Use when a model needs to be served - engine and quantization chosen, docker or Kubernetes deployment prepared as a gated runbook, or the endpoint measured with the TTFT/TPOT/ITL/goodput protocol - and the user wants the process, not an ad-hoc command guess.

Strong routing signals: `inference-serving`, `inference serving`, `serve this model`, `serve the model`, `model serving`, `serving endpoint`, `vllm`, `llama.cpp`, `llama cpp`, `serve with vllm`, `deploy vllm`, `vllm deployment`, `serving benchmark`, `benchmark the endpoint`, `prefix caching benchmark`, `gguf quantization`, `which quantization`, `모델 서빙`, `모델 서빙해줘`, `모델 배포해서 서빙`, `서빙 벤치마크`, `vllm 배포`, `vllm 서빙`, `추론 서버 띄워줘`, `모델 띄워줘`

Catalog Metadata

Category: `operations` Phase: `inference-serving` Hermes role: `operator` Quality tier: `observed-command-gated` Reasoning demand: `light`

Quality bar:

• Decide before deploying: engine from the situation table (vLLM for multi-user NVIDIA APIs, llama.cpp for CPU/Apple Silicon/edge, TensorRT-LLM only with ops budget), quantization to match (AWQ/GPTQ/FP8 vs the GGUF ladder with `Q4_K_M` default), tensor parallel a power of two.
• Deploy as the gated runbook: docker's three load-bearing flags (`--ipc=host`, HF cache mount, `HF_TOKEN`) or the Kubernetes five-step (secret gate, existing-deployment gate, apply, rollout+readiness verify, summary+smoke); the port invariant touches four places or it did not change the port.
• Troubleshoot from the symptom table first - slow TTFT to prefix caching/chunked prefill, OOM to gpu-memory-utilization/max-model-len/quantization - before inventing flags.
• Measure with the protocol: TTFT/TPOT/ITL/E2EL as mean/median/P99, goodput against an explicit SLO, one load shape per run, results saved with metadata; the full contract is `omh-inference-serving/references/serving-bench.md`.
• Report observed-only: each runbook step is prepared until its command's exit status and output are seen.

Handoff policy:

Keep engine/quantization decisions, runbook preparation, and benchmark design in Hermes; the commands run through the operator's terminal with observed evidence, and repository changes (deploy manifests, benchmark harnesses) are coding work for the selected executor lane. A runbook or benchmark plan is prepared_not_observed until its commands' results are seen.

Required inputs:

• the model id(s) and whe