/audit-my-experiment

name: audit-my-experiment
description: Use when a measured result — an experiment, a forecast, a number that must tie out — is about to drive a decision; the validity checks run before the decision does. An experiment / A-B test / causal result is about to drive a decision - ship, roll out, shift budget - including when someone just wants the win written up. COMPUTES the validity checks a consumption read eyeballs past - sample-ratio mismatch, peeking, multiple comparisons, power/MDE, Simpson's, novelty, metric-vs-proxy - via a tested kit on the summaries in hand; missing data becomes the exact check you run and paste back. Detects: "validate this experiment", "is this A/B result real", "should we ship this test", "did the test pass", "write up our experiment win", "should we roll this out". Within this family: a forecast / time-series projection is audit-my-forecast. Boundary: the SQL behind the metric is review-my-query; unvalidated source premises are audit-my-assumptions. Read-only on your data: never connects to a live system or raw data.
allowed-tools: Read, Write, Bash

audit-my-experiment

The colleague who runs the checks before you ship the result: computes the validity tests you'd otherwise eyeball, tells you what's broken and what can't be verified yet, and never blesses a number it didn't check.

When to use

Fire when an experiment / A-B / causal result is heading to a decision — *even under a consumption ask* ("write up our win", "should we ship"). Switch into audit-mode and validate before packaging. Do NOT fire to write up an already-validated result (`brief-my-findings`), rehearse defending it (`defend-my-number`), review ONE code object (`review-my-query`), diagnose why ONE number moved (`triage-my-number`), audit a whole KB (`kb-reconcile`), or define a metric (`kpi-contract`).

The trap this exists to beat

A capable model reads an experiment result and writes the win — and it does the analytical part well: it recognizes Simpson's paradox if segments are shown, catches a narrated novelty story, flags a named peeking admission. Then it does the wrong thing with the checks it should compute. Its instinct is to eyeball the split ("looks roughly 50/50"), glance at p=0.03 and call it significant, and note that "nothing else hit significance" as reassurance. It writes the win under a consumption ask and ships it. The discipline it skips: switch OUT of answer-mode into audit-mode and COMPUTE the checks rather than eyeballing them.

Proven: under the consumption framing ("write up our win"), a cold model shipped a 0.56% SRM — a χ²≈7.8, p≈0.005 — dismissed as "expected noise at scale." The check was never run. The same model waved a peeked p<0.05 through without applying a sequential threshold. Both failures are invisible to a reader; only computation catches them.

The loop

1. **Switch to audit-mode + set the target.** Recognize an experiment/A-B/causal result headed for a decision, even under a consumption ask. Pin the claim & decision riding on it, the **identification strategy (RCT / DiD / other non-RCT — geo, pre/post, IV, synthetic-control)**, primary metric, arm counts, the stopping story, the metric family, **and the minimum-meaningful-effect (MME) — the smallest effect that would change the decision (cost/benefit breakeven or launch bar); elicit it, or mark `materiality-unverified` (never invent it).** 2. **Inventory in-hand vs needs-data.** Separate checks computable from the summary numbers given (SRM, two-proportion z/CI, multiplicity, power/MDE) from checks needing data not on hand (per-day assignment logs, pre-registration, missing segment cuts). 3. **Run the computable checks with the kit — don't eyeball.** Execute `references/experiment_checks.py` with the provided numbers; report each computed statistic. SRM chi-square runs on ANY split. 4. **Run the full validity taxonomy (the engine).** `references/validity-taxonomy.md`: design / inference / interpretation layers. Comprehensive thinking, lean output — record what bites. 5. **Write the check for anything unverifiable.** Exact query/script; mark `unverified — needs paste-back`. On a pasted run, reconcile (the run wins). Never bless what you can't compute. 6. **Grade + gate.** Blocking / Latent / Advisory, each with computed evidence + fix direction. A Blocking validity defect gates the ship/brief decision. **Materiality rides as its own verdict line — `material` / `immaterial` / `straddles-MME` / `materiality-unverified` (run `classify_materiality`) — carried into the handoff; it does NOT gate (a valid experiment can be immaterial), but a `ship-ready` result is never written up as a material win without it.** 7. **Emit + route.** Write `experiment-audit.md`; if `ship-ready`, hand off to `brief-my-findings` / `defend-my-number`. KB composition per `references/experiment-audit.md`. Then stop.

The signature output

A graded `experiment-audit.md` with a *computed* statistic per check (not an eyeball verdict). Every applicable check ends `pass` / Blocking / Latent / Advisory / `unverified`; no check is silently skipped. The point is the Blocking validity defects — what gates the ship decision — plus the explicit list of checks that need a paste-back to clear. Template and KB composition rules live in `references/experiment-audit.md`.

Running the checks

Invoke the tested kit `references/experiment_checks.py` via `Bash` with the user's summary numbers — never hand-compute. Functions: `srm_chisquare`, `two_prop_z`, `multiplicity_correct`, `power_mde`, `peeking_flag`, `chi2_sf`. The module has no CLI — import and call the functions (e.g. `python3 -c "import experiment_checks as ec; print(ec.srm_chisquare([500000, 502800]))"` from the references dir).

If `Bash` is unavailable (Read/Write-only deployment), degrade gracefully: write the exact check for the user to run and paste back. Mark every computable check `unverified — needs paste-back` until the run is provided. The audit still runs; it ju