/audit-my-forecast

name: audit-my-forecast
description: Use when a measured result — an experiment, a forecast, a number that must tie out — is about to drive a decision; the validity checks run before the decision does. A forecast / projection / time-series prediction is about to drive a plan - demand, capacity, revenue, headcount, budget - or someone wants the forecast written up or "trusted". Checks the four temporal-validity failure modes - leakage, backtest validity, interval honesty, drift - computing coverage, accuracy-vs-naive, and error trend from the actuals-vs-predicted in hand via a tested kit. Detects: "validate this forecast", "is this projection any good", "should we plan against this", "the backtest looks great - ship it", "did the model hold up", "our forecast says X". Within this family: a controlled A/B / causal-effect result is audit-my-experiment. Boundary: the feature-build code is review-my-query. Read-only: never fits, refits, tunes, or re-forecasts a model, and never connects to a live series.
allowed-tools: Read, Write, Bash

audit-my-forecast

The colleague who checks whether the forecast will actually hold before you plan against it: computes the coverage and skill tests you'd otherwise eyeball, names the leakage and backtest traps that make a backtest lie, and never blesses a projection it didn't check.

When to use

Fire when a forecast / projection / time-series prediction is heading into a plan — *even under a consumption ask* ("write up the forecast", "should we plan against this"). Switch into audit-mode and validate before the plan rides on it. Do NOT fire for a controlled A/B / causal-effect result (`audit-my-experiment`), to diagnose why ONE production number moved (`triage-my-number`), to review the SQL/feature-build code as text (`review-my-query`), to pin the target metric's definition (`kpi-contract`), or to write up an already-validated forecast (`brief-my-findings`). **This vs. `audit-my-experiment`:** experiment audits a *controlled causal test* ("is this lift real" — SRM, peeking, power); this audits an *extrapolation into the future* ("will this projection hold" — leakage, backtest, intervals, drift). Same shape (audit → gate → `*-audit.md`), different question.

The trap this exists to beat

A capable model reads a forecast with a great-looking backtest and writes "the model is accurate, plan against it" — and the backtest is lying. Its instinct is to trust the reported error, eyeball the fan chart, and never ask how the split was made. The four silent failures it skips: **leakage** (a feature that wasn't knowable at forecast time, or scaling computed over the test window, makes the backtest far too good); **backtest design** (a random K-fold shuffles time and destroys causal order; no naive baseline means "low MAPE" is meaningless); **interval honesty** (a point forecast shipped as certain, or a "95%" band that actually covers 60%); **drift** (the model was accurate at fit and has silently decayed since). All four are invisible in the reported accuracy; only structural inspection plus computing coverage/skill/trend on the actuals-vs-predicted catches them. This skill switches into audit-mode and does exactly that.

The loop

1. **Switch to audit-mode + set the target.** Pin the forecast claim, the plan/decision riding on it, the horizon, and what's in hand: the actuals-vs-predicted series? the backtest setup? the stated intervals? the feature list? 2. **Inventory in-hand vs needs-data.** Computable from supplied actuals-vs-predicted (interval coverage, accuracy-vs-naive, error trend) vs needs-data / narrative (the train/test split design, each feature's as-of availability). 3. **Run the computable checks with the kit — don't eyeball.** Execute `references/forecast_checks.py` on the provided numbers; report each computed statistic. 4. **Run the full temporal-validity taxonomy (the engine).** `references/temporal-validity.md`: leakage / backtest / interval / drift. Comprehensive thinking, lean output — record what bites. 5. **Write the check for anything unverifiable.** Exact script/query; mark `unverified — needs paste-back`. On a pasted run, reconcile (the run wins). 6. **Grade + gate.** Blocking / Latent / Advisory, each with computed-or-structural evidence + fix direction. A Blocking temporal-validity defect gates the plan. 7. **Emit + route.** Write `forecast-audit.md`; if `trustworthy`, hand to `brief-my-findings` / `defend-my-number`. KB composition per `references/forecast-audit.md`. Then stop.

The signature output

A graded `forecast-audit.md` with a *computed* statistic where computable (coverage, skill-vs-naive, error trend) and a *structural* finding where not (leakage, split design) — every applicable check ends `pass` / Blocking / Latent / Advisory / `unverified`; no check silently skipped. The point is the Blocking temporal-validity defects — what gates the plan — plus the explicit list of checks needing a paste-back. Template + KB composition in `references/forecast-audit.md`.

Running the checks

Invoke the tested kit `references/forecast_checks.py` via `Bash` on the user's supplied actuals-vs-predicted — never hand-compute, never fit a model. Functions: `interval_coverage`, `mape_vs_naive`, `error_trend`. Import and call (e.g. `python3 -c "import forecast_checks as fc; print(fc.interval_coverage(actuals, lows, highs))"` from the references dir). If `Bash` is unavailable (Read/Write-only deployment), degrade: write the exact check for the user to run and paste back; mark each computable check `unverified — needs paste-back`.

Bright lines (the teeth; inherits groundwork's read-only line)

• **Never fit, refit, tune, re-forecast, or pick the model.** You audit a forecast; you do not build one. ("Let me just retrain to compare" → stop; that's the modelling lane.)
• **Never connect to a live system or raw series.** Compute on the supplied actuals-vs-predicted summaries only; for anything else, write the exact check and requi