/experiment

name: experiment
license: MIT
description: >-
  Automated optimization loop with scalar fitness function. Proposes changes in
  isolated worktrees, measures against a fixed baseline, and retains verified
  candidates locally. Supports convergence detection and diminishing returns.
user-invocable: true
auto-trigger: false
trigger_keywords:
  - experiment
  - optimize
  - try
  - A/B
  - measure
last-updated: 2026-10-15

/experiment — Metric-Driven Optimization Loop

Identity

Coordinate bounded metric-driven experiments in isolated checkouts, retaining candidates locally until the declared verification and independent review pass.

Orientation

When to Use

• When you want to optimize a measurable metric (bundle size, error count, test coverage, FPS)
• When you have a clear hypothesis but aren't sure which of several approaches wins
• When manual A/B testing would be too slow or error-prone
• NOT when the goal is subjective ("make it feel better") — the metric must be a number

Inputs

The user provides three things: 1. **scope**: Files to modify (glob pattern, e.g., "src/api/**/*.ts") 2. **metric**: Project command that outputs a single number (e.g., `node tools/bundle-bytes.cjs`) 3. **budget**: Iteration cap (default: 5) or time cap (e.g., "10 minutes")

Ask only for missing inputs that cannot be resolved from the request or project. The metric MUST exit successfully and output a single finite number to stdout.

Protocol

Step 1: BASELINE

1. Read project instructions and inspect the checkout, index, untracked files, and active worktrees. Record the source commit and dirty-file hashes/status. Leave user work in place: never stash, reset, clean, switch branches, or commit unrelated changes to make an experiment possible. 2. Create a dedicated worktree at an explicit commit, or a separate local clone at that commit. Verify its path, HEAD, and clean status before running commands. If the requested baseline includes authorized working changes, copy only those changes into the isolated checkout, preserving deletions and file contents; verify their hashes and commit the snapshot there. Leave the original index and files untouched. If the intended snapshot is unclear, clarify before copying. 3. Before iterations, write the evaluation plan in the experiment report: source and snapshot commits, allowed source scope, metric command and direction, timeout, repeat/noise policy, and required typecheck, regression and behavioral checks. Record any inapplicable check and its reason now, not after a failure. Ask about direction only if ambiguous. A scalar is a proxy, not acceptance. 4. Freeze the evaluator: record hashes for the metric implementation, test assertions, fixtures/expected outputs, and relevant command/config files. These are outside candidate edit scope, even if a broad glob includes them. Keep the original incumbent commit, evaluator and plan fixed for the run. 5. Pin one file-byte representation before measuring, including checkout filters and line endings. Compare baseline and evaluator hashes with their committed blobs; for commit-based evaluation, materialize matching bytes in the isolated checkout. Use the same representation for every candidate and independent check; a clean Git status alone does not establish byte identity. 6. Measure the baseline and run the declared checks in isolation. Record commands, exit codes, values and evidence paths. An invalid baseline or required-check failure holds optimization until resolved; do not silently waive the check.

Run commands from the intended isolated checkout. Bind `CLAUDE_PROJECT_DIR` for each child process to that checkout so timeout/telemetry helpers cannot write into the user's source checkout. Use the installed timeout helper (in Citadel itself, `node scripts/run-with-timeout.js 300 <command> [args...]`).

Step 2: ITERATE

For each iteration (up to budget):

1. **Create isolation**: Use a fresh worktree/clone from the original incumbent or the last verified retained candidate; record that parent commit. Give the worker the fixed plan, scope, evaluator exclusions and previous attempts. Verify actual isolation rather than relying on a branch name or a spawn option alone. 2. **Propose change**: Modify only allowed source files. Never change the metric, tests, expected outputs, exclusions or thresholds to earn a pass. If the evaluator needs repair, stop this run and establish a new baseline and plan. 3. **Inspect and measure**: Check the full diff and evaluator hashes first. Reject scope violations or evaluator changes, even if they improve the number. Run the unchanged metric and declared checks with bounded timeouts and the same setup as the incumbent. Keep raw results, candidate commit and evaluator identity. 4. **Review**: For an improving candidate, obtain independent review of the exact diff, scope, behavior/regressions and evidence. The proposer cannot approve its own result. A summary-only HANDOFF check is insufficient. Review unavailability, stale evidence, timeout or an incomplete required check means HELD, not passed. 5. **Decide**:

• **KEEP**: Improved under the declared repeat/noise policy, with every required

check and independent review passed for this exact candidate. Retain its local commit; it may seed the next iteration. Do not merge into the user's checkout.

• **DISCARD**: Same/worse, failed required check, scope violation or evaluator

tampering. Log why; preserve evidence before removing only experiment-owned disposable worktrees. Never clean another agent's or the user's worktree.

• **HELD**: Missing/unknown evidence or review. Preserve the local candidate and

reason; it cannot seed a subsequent iteration or authorize promotion. 6. **Log**: Record value, delta against both the original baseline and the previous retained best, verdict, change, commi