/test-codex-locally

name: test-codex-locally
description: Run the Weave router locally in docker compose and drive it with `codex exec` to reproduce and verify routing/translation/marker behavior for Codex's Responses API path. Use when verifying a router fix end-to-end for Codex, reproducing a prod Codex routing bug, confirming routing-marker / force-model / subscription-passthrough behavior, or testing a `/force-model` route — without touching the user's global Codex config.

Testing the router locally with Codex

> For an **automated** pre-merge regression net (fixture-driven, asserts caching/streaming/decision-headers against real Anthropic), run `make smoke` — see [docs/SMOKE.md](../../../docs/SMOKE.md). This skill is the interactive counterpart for the **Codex CLI** (`codex exec`): stand the stack up by hand and drive it via a throwaway `CODEX_HOME` so `~/.codex/config.toml` is never edited.

Stand up the router in docker compose, point a one-off `codex exec` at it via `CODEX_HOME`, and read the local server logs to confirm behavior. Two upstream modes: the **real** provider API (needs a working key + credits, and for native GPT models a ChatGPT OAuth login) or a **mock** upstream that emits an exact SSE shape (deterministic, no credits).

The sibling skill [test-claude-locally](../test-claude-locally/SKILL.md) is the Claude Code (`claude -p`) counterpart. Stack bring-up and seeding are identical; only the client driver differs.

Critical gotchas (read first)

• **Never edit `~/.codex/config.toml` or `~/.codex/auth.json`.** Those are the user's live Codex install (prod Weave Router + ChatGPT OAuth). Redirect with a throwaway `CODEX_HOME` directory that contains its own `config.toml` (and a *copy* of `auth.json` when ChatGPT OAuth is required). `CODEX_HOME` is how Codex finds config; there is no `--settings` flag equivalent to Claude Code.
• **`requires_openai_auth = true` is required.** Codex 0.149 hangs or never issues a `/v1/responses` request if the custom provider uses `requires_openai_auth = false` or `env_key`. Copy `~/.codex/auth.json` into the throwaway `CODEX_HOME` so Codex can attach the ChatGPT JWT + `ChatGPT-Account-ID`. The mock-router experiment that dropped OAuth never even hit the mock.
• **Codex always sends a ChatGPT JWT + `ChatGPT-Account-ID`.** The local router will classify that as a Codex subscription (`codexResponsesRequest` → native Responses passthrough for OpenAI decisions). That is the path this skill is usually exercising (routing markers on verbatim GPT frames). To force the prepaid/BYOK translation path instead, flip `subscription_routing_disabled=true` on the seeded installation (see step 3).
• **`codex exec` is one-shot.** A standalone `/force-model` call does not persist to the next `codex exec` (new session id every time, unless you `codex exec resume`). Put ` /force-model <model>` as the first line of the SAME prompt. The leading space is load-bearing — Codex consumes unknown slash tokens as local commands; a leading space makes it a normal user message the router can parse.
• **Pin via `config.toml`, not `codex -c`.** On Codex 0.149.1, `-c 'model_providers.weave-local.http_headers."x-weave-force-model"="…"'` did **not** merge into the outgoing request (no `x-weave-force-model applied` in server logs; the scorer served something else). Put `"x-weave-force-model" = "<id>"` on the same `http_headers` table in the throwaway `config.toml` (step 4). Confirm with `docker compose logs server | grep 'x-weave-force-model applied'` before drawing conclusions.
• **The router ignores the request's `model` field** for routing (cluster scorer / pins win). Codex's `-m gpt-5.5` only changes what Codex *requests*; `/force-model` or `x-weave-force-model` is the only way to pin a specific routed model. Native GPT family (`gpt-5.6-sol` / `terra` / `luna`, plus whatever `-m` Codex defaulted to) still goes through the scorer unless forced.
• **No GNU `timeout` on macOS.** Drive `codex exec` directly. If a run hangs (`Reading additional input from stdin...` with no banner), Codex is waiting on a TTY/stdin — do not pipe into it, and make sure `-s read-only` (or `--dangerously-bypass-approvals-and-sandbox`) is set.
• **Docker may not be running.** `docker compose` fails with a `unix://…/.docker/run/docker.sock` connect error if Desktop is stopped. `open -a Docker` and wait for `docker info` before step 1.
• **Other agents in the same workspace can `docker compose down` and delete `/tmp` scratch.** If health suddenly 404s or `CODEX_HOME` vanishes mid-run, re-up the stack and reseed — do not reuse a key you can no longer `/validate`.
• **Session key is `apiKeyID` + first user message.** Reusing the same curl/prompt body on the same key reuses the pin slot. `routingMarkerFor` then returns empty when `PriorServedModel == ServedIdentity()` (sticky same-model). For first-turn marker tests, seed a **new** key or change the first user text. Isolated `/validate` first: `curl -sS -o /dev/null -w '%{http_code}\n' -H "X-Weave-Router-Key: $KEY" http://localhost:8080/validate` must be `200`.
• **Native GPT passthrough + tool-only emits a synthetic badge item.** `SetPassthroughBadge` rewrites existing `response.output_text.*` events and, when a ChatGPT-subscription turn has no assistant text, inserts a leading assistant message item before native tool/reasoning output. Verify the badge in the Responses SSE/JSON; the Codex TUI can bury the first-turn line under tool chatter.
• **`GET /v1/models` 501 from a mock is fine.** Codex probes `<base_url>/models` first, logs an HTML 501, then continues to `POST /v1/responses`. The real local router implements `GET /v1/models` as Anthropic passthrough, so this only shows up against a Python mock.
• **Port 8085 conflict.** The monorepo's pubsub emulator may already own host port 8085. Drop the router's host binding with a `docker-compose.override.yml` (see workflow). The server still reaches the emulator over the compose network.
• **No credits / no key =