/calibrate

name: calibrate
description: Calibration testing for agents and skills. Generates synthetic problems with known outcomes (quasi-ground-truth), runs targets against them, measures recall, precision, confidence calibration — reveals whether self-reported confidence scores track actual quality.
argument-hint: '[<scope>...] [--fast | --full] [--ab-test | --apply] [--skip-gate] [--local] [--keep "<items>"]'
effort: high
disable-model-invocation: true
allowed-tools: Read, Write, Bash, Agent, Glob, TaskCreate, TaskUpdate, TaskList, AskUserQuestion

<objective>

Validate agents and skills by measuring outputs against synthetic problems with defined ground truth. Primary signal: **calibration bias** — gap between self-reported confidence and actual recall. Well-calibrated agent reports 0.9 when it finds ~90% of issues. Miscalibrated: reports 0.9, finds 60%.

Calibration data drives improvement loop: systematic gaps: instruction updates; persistent overconfidence: overconfidence checks in the target's `<antipatterns-to-flag>`.

NOT for: static routing overlap analysis (use /foundry:audit); manually reviewing skill output quality (use /develop:review (requires `develop` plugin)).

</objective>

<inputs>

• **$ARGUMENTS**: parse `--flags` first, then resolve remaining tokens as scope targets

**Flags** (order independent):

• `--fast` — 3 problems per target (default when neither pace flag passed)
• `--full` — 10 problems per target; mutually exclusive with `--fast`
• `--ab-test` — also run `general-purpose` baseline and report delta metrics; requires benchmark (default `--fast` if no pace flag); mutually exclusive with `--apply`
• `--apply` — prepare proposals for application: with `--fast`/`--full`: benchmark then review the concrete edits; without pace flag: skip benchmark, select the most recent proposal per target; application always requires the Step 6 approval; mutually exclusive with `--ab-test`
• `--skip-gate` — suppress fan-out and follow-up gates; for programmatic callers; never bypasses Step 6 edit approval
• `--local` — read rules (`rules` target), mode files and the cross-plugin availability check from source tree (`plugins/*/`) instead of installed plugin cache; agent/skill targets come from source tree in both modes (`resolve_agent_file.py`); for plugin-dev workflows where local edits aren't yet installed; sets `LOCAL_MODE=true` in all pipeline spawns

**Mutual exclusion validation** (check before any work):

• `--ab-test` + `--apply` together: hard error: "`--ab-test` and `--apply` are mutually exclusive. Pass one or neither."
• `--fast` + `--full` together: hard error: "Pass `--fast` or `--full`, not both."
• `--ab-test` without pace flag: default `--fast` silently (no error)

**Unsupported flag check** — after all supported flags extracted (`--fast`, `--full`, `--ab-test`, `--apply`, `--skip-gate`, `--local`, `--keep`), scan `$ARGUMENTS` for remaining `--<token>` tokens. Found: invoke `AskUserQuestion` with question text `` ! Unknown flag(s): `--<token>`. Supported: `--fast`, `--full`, `--ab-test`, `--apply`, `--skip-gate`, `--local`, `--keep`. `` — (a) **Abort** (stop, re-invoke with correct flags) · (b) **Continue ignoring** (skip unknown flags, proceed). On Abort: stop.

**Legacy positional tokens** (`ab`, `apply`, `fast`, `full`) — **hard error**: print migration hint, stop. Example: "`ab` removed — use `--ab-test` flag: `/foundry:calibrate curator --ab-test`."

**Scope tokens** (positional, space-separated — defaults to `all`):

• `all` — all agents + relevant skills + routing + communication + all rules
• `agents` — all agents only (full agent list in `modes/agents.md`)
• `skills` — calibratable skills only (`/foundry:audit` and others per `modes/skills.md`; `/oss:review` (requires `oss` plugin) excluded — requires live GitHub PR)
• `routing` — routing accuracy test: measures how accurately `general-purpose` orchestrator selects correct `subagent_type` for synthetic task prompts (not per-agent quality benchmark; included in `all`)
• `communication` — handover + team protocol compliance: runs `foundry:curator` against synthetic agent responses and team transcripts with injected protocol violations (missing JSON envelope, missing `summary`, AgentSpeak v2 breaches); included in `all`
• `rules` — rule adherence test: for each global rule file (no `paths:`) and each path-scoped rule when matching file is in context, generates synthetic tasks that should trigger rule's key directives, measures whether `general-purpose` agent with rule loaded correctly applies them; reports rules that are ignored, misapplied, or redundant; included in `all`
• `plugins` — all agents + calibratable skills from all `plugins/*/` directories (union of all plugin-namespaced agents and calibratable skills)
• `<plugin-name>` — **tier 2**: plugin name or directory name (e.g. `oss`, `foundry`, `research`, `develop`, `codemap-py`); resolve `plugins/<name>/` first, then `plugins/cc_<name>/`, then a plugin root whose manifest `name` matches (e.g. `bridge` → `plugins/bridge_cc-codex/`); calibrates that root's manifest-declared agents + calibratable skills
• `<agent-name>` — **tier 3**: single agent (e.g., `foundry:sw-engineer`); also accepts bare name (e.g. `sw-engineer`) and resolves via `plugins/*/agents/<name>.md`
• `/foundry:audit` — single skill (pass any calibratable skill name; `/oss:review` (requires `oss` plugin) accepted but excluded per `modes/skills.md`)
• Multiple scope tokens — space-separated; calibrates union of resolved targets: `oss research`, `agents skills`, `curator shepherd`; each token resolved through same tier hierarchy as `/foundry:audit` scope tokens (reserved keywords first, then plugin-dir lookup, then agent/skill file search)

Every invocation surfaces report: benchmark runs print new results; `--apply` without pace flag prints saved report from last run before applying.

</inputs>

<constants>

• FAST_N: 3 problems per tar