/vllm-sr-agent-operations

name: vllm-sr-agent-operations
description: Install, configure, and verify vLLM Semantic Router on a Docker host (CPU, AMD, or NVIDIA GPU) or a Kubernetes cluster, then operate it through its CLI and Router API. Use for a first install, an upgrade, routing verification, configuration changes, recipe tuning, and single-model/MoM evaluation.

vLLM Semantic Router: install, configure, verify

Run the steps in order. Each one says what to run, what success looks like and what to do otherwise, and each is safe to run again.

The Router serves an OpenAI-compatible API and picks a model for each request. It does not run the models that answer users; those are the user's endpoints, such as vLLM, Ollama or a hosted API. `vllm-sr serve` runs a stack in Docker: the Router (inference on port 8899, management API on 8080), the Dashboard (8700), the model runtime for the Router's own classifiers, and supporting services.

Ground rules

• Keep secrets in environment variables, never in YAML, command arguments,

logs or your report.

• Don't stop, restart or replace what you didn't start (containers, a running

`vllm-sr` stack, services on its ports) without asking.

• Ask before system-wide changes: installing packages, adding the user to a

group, publishing a port beyond loopback.

• Never run `vllm-sr serve` without a complete `--config`. Without one it

starts the Dashboard's setup mode and waits for a person indefinitely.

• An HTTP 200 alone is not success. Each check below names what the output

must contain.

1. Preflight

These commands change nothing. Run them and keep the results.

uname -sm
docker info --format '{{.ServerVersion}}'
python3 -c 'import sys; print(sys.version.split()[0])'
python3 -m ensurepip --version
df -h "$HOME" | tail -1
ls /dev/kfd /dev/dri/render* 2>/dev/null
nvidia-smi -L 2>/dev/null
vllm-sr --version 2>/dev/null && vllm-sr status

| Check | Pass | Otherwise | | --- | --- | --- | | Docker | prints a version | "permission denied": ask the user to add their account to the `docker` group. Not installed: ask before installing it. Podman works too: add `--container-runtime podman` to `vllm-sr` commands. | | Python | 3.10 or newer, and `ensurepip` prints `pip …` | Ubuntu and Debian: with the user's OK, `sudo apt-get install -y python3-venv`. Without it the installer stops at "ensurepip is not available". | | Disk | 10 GB free; 30 GB for a GPU platform | Ask the user. | | `vllm-sr` | not installed, or `status` says `Not running` | A stack is already running. It is the user's: find its directory (below), verify it with step 6 using its own decision and model names, and ask before changing it. |

Then check the host ports the stack publishes:

bash -c 'for p in 8899 8080 8700 8090 9190 50051 6379 3000 9090 16686; do
  (exec 3<>/dev/tcp/127.0.0.1/$p) 2>/dev/null && echo "port $p is in use"; done'

Nothing printed: pass. Otherwise ask the user. A second stack can run beside the first: see [Deployment](https://vllm-sr.ai/install/agent/vllm-sr/references/deployment-loop.md#stack-identity).

A running stack's directory is the parent of the `.vllm-sr` directory its Router mounts:

docker inspect vllm-sr-router-container \
  --format '{{range .Mounts}}{{println .Source}}{{end}}' | grep -m1 '/\.vllm-sr$'

2. Choose the path

Decide each row and tell the user what you chose.

| Decision | Default | Otherwise | | --- | --- | --- | | Channel | stable, when it has this skill's commands (step 3); else dev | The user names a version or channel. | | Platform | `auto`: inspect the execution target | `/dev/kfd` and a `/dev/dri/render*` node exist: `--platform rocm` (AMD Instinct MI300X and MI325X are validated). `nvidia-smi -L` lists a GPU and `docker info` lists an `nvidia` runtime (the NVIDIA Container Toolkit): `--platform cuda` (works, not yet validated). macOS: always `cpu`. | | Gateway | standalone (no flag): the Router serves the API itself | `--gateway extproc` puts Envoy in front. Use it only when the user needs Envoy's rate limiting, mTLS, JWT or OIDC, advanced route matching, or an Envoy-based gateway they already run. | | Target | Docker on this machine | The user asks for a Kubernetes cluster: write the configuration (step 4), then follow [Deployment](https://vllm-sr.ai/install/agent/vllm-sr/references/deployment-loop.md#kubernetes) instead of steps 5 and 6. | | Model | the endpoint the user gave | None given: look for one on this host (below) and confirm it with the user. None at all: ask. For a local trial, offer [Ollama](https://vllm-sr.ai/docs/installation/ollama). |

Look for a model server on this host:

curl -s -m 3 http://127.0.0.1:11434/api/tags     # Ollama: its model names
curl -s -m 3 http://127.0.0.1:8000/v1/models     # vLLM or another OpenAI-compatible server

The Router runs in a container and reaches a server on this host as `host.docker.internal:PORT`, so that server must listen beyond 127.0.0.1 (Ollama: `OLLAMA_HOST=0.0.0.0:11434`; vLLM: `--host 0.0.0.0`). Check it the way the Router will, here for Ollama:

docker run --rm --add-host=host.docker.internal:host-gateway \
  curlimages/curl:8.12.1 -s -m 5 http://host.docker.internal:11434/api/tags

Pass: the same model list.

3. Install the CLI

curl -fsSL https://vllm-sr.ai/install.sh | \
  bash -s -- --channel stable --mode cli --runtime skip --no-launch
export PATH="$HOME/.local/bin:$PATH"
vllm-sr --version
vllm-sr serve --help | grep -q -- '--data-parallel-size' && echo "current" || echo "predates this skill"

`--mode cli --runtime skip --no-launch` installs only the CLI, into `~/.local/share/vllm-sr` with the launcher `~/.local/bin/vllm-sr`. Later commands need that `PATH`; if your shell doesn't keep it between commands, repeat the `export`. Without those three options the installer also starts a setup-mode stack and returns, for a user who would rather connect models in the Dashboard; then hand over its URL instead of