/tune-loop

name: tune-loop
description: The tunelab capstone — drives a self-improving AI system. Use when the user wants a deployed model/cascade/workflow to keep getting better from feedback, run champion/challenger experiments to discover the best architecture, set up a data flywheel with retrain triggers, or automate "is a new model better than what we ship?" decisions. Orchestrates tune-decide/tune-data/tune-train/tune-eval as a Monitor-Analyze-Plan-Execute loop. The system may be one model, a cascade, an agent, or a deterministic multi-model workflow — tune-loop experiments across architectures and fine-tuning methods and promotes only what beats a pre-registered bar on a fresh eval slice.

tune-loop — the self-improving system

The other four skills build *one* model. tune-loop closes the loop: serve → log → collect feedback → curate → experiment across architectures × methods → **promote only what measurably beats the champion** → repeat. The "AI system" stops being a model and becomes the loop; what sits inside (a single model, a cascade, an agent, a deterministic workflow) is just the current champion, which the loop is free to replace when evidence says so.

This is champion/challenger from classical MLOps, generalized to compound-AI architecture search, driven by a Monitor–Analyze–Plan–Execute (MAPE) control loop. It deliberately reverses tunelab's v1 "no autonomous hill-climbing" non-goal — and the disciplines below are what make the reversal teaching-grade rather than AutoML slop.

Step 0 — read the system state before anything

On invocation, read the project's `system/` dir:

• **`descriptor.json`** — the current champion architecture (see schema below).
• **`predictions.jsonl`** — the append-only flywheel log (tune-data's `flywheel.py` schema).
• **`EXPERIMENT-LOG.md`** + **`rounds/*/`** — prior rounds, promotions, and the eval slices each

consumed. Never reuse a consumed slice.

The system descriptor (architectures as data)

A small versioned JSON makes architectures enumerable, comparable, reproducible:

{ "version": 3, "kind": "cascade",
  "components": [
    {"id": "t1", "model": "lr", "train": "embeddings+logreg", "calibrate": "isotonic"},
    {"id": "t2", "model": "qwen3-4b", "train": "qlora-sft", "conf": "token-margin"},
    {"id": "t3", "model": "frontier", "prompt": "knn-fewshot"}],
  "routing": "t1 ->[cal_conf<0.43] t2 ->[cal_conf<0.60] t3",
  "thresholds": {"t1": 0.43, "t2": 0.60} }

`kind` ∈ {single, cascade, workflow}. Examples: `single: FT-SLM(qwen3-4b, sft)` · `workflow: injection-gate → router → {cheap, frontier}`. The descriptor is the unit the loop mutates and the eval adjudicates.

The MAPE round (the crank)

Monitor + Analyze — `flywheel.py status`

Read the prediction log; report **audit-slice accuracy** (the honest served estimate — never the biased feedback pile), confidence drift (PSI), per-tier coverage, and which retrain triggers fire. Triggers: `--min-new-labels`, `--drift-psi`, `--accuracy-floor`. No trigger → hold, and lengthen the next check interval (the system self-paces toward stability).

Plan — generate challengers from the portfolio

On a fired trigger, take a snapshot, curate it (`flywheel.py plan` — dedupe, resolve conflicts by recency, **exclude every holdout id**), and propose a **challenger set**. Search is staged to stay bounded: 1. **Architecture family first** — single vs cascade vs workflow; add/drop a tier. 2. **Then one refinement per round** — LoRA vs DoRA, rank, an RLVR round, an embedding upgrade, a threshold re-fit. One factor at a time so a win is attributable. Score cheap proxies first: **offline counterfactual composition** (`cascade_compose.py`) ranks threshold/architecture choices from existing per-tier predictions before any training. Train only the survivors, within a declared compute/$ budget. **STOP — get human sign-off on the budget and the challenger list before spending** (delegable, like a standing "continue" arrangement).

Execute — train + adjudicate (champion vs challengers)

tune-train trains the survivors; tune-eval scores champion and challengers on a **fresh, never-used eval slice** against the pre-registered promotion bar. **STOP — promotion is a human checkpoint.** Promote (write a new descriptor version) only if a challenger BEATS the bar; ties retain the champion (cost/latency tiebreak). Append the round — challengers, scores, decision, the slice consumed — to EXPERIMENT-LOG.md either way.

**Wiring the adjudication inputs (explicit — there is no magic step):** score each model with `eval_classifier.py --predictions preds.jsonl --json champion_eval.json` (and again for the challenger) — `--json` emits exactly the `{accuracy, ...}` file `promote.py` reads. For the **first** promotion, before any flywheel exists, there is no `system/` dir yet — seed one: write a v1 `system/descriptor.json` describing the current champion and a `system/challenger.json` describing the contender (its run id / adapter path), then:

uv run .../tune-loop/scripts/promote.py \
  --champion champion_eval.json --challenger challenger_eval.json \
  --bar <pre-registered> --min-margin 0.02 --metric accuracy \
  --slice-id <fresh-slice-id> --ledger system/consumed_slices.txt \
  --descriptor-in system/descriptor.json --challenger-descriptor system/challenger.json \
  --descriptor-out system/descriptor.json

On PROMOTE the **challenger** descriptor becomes the new champion (so the record says what won — the tuned adapter — not just a bumped version on the old one).

The three failure modes this design must own

1. **Feedback bias** — feedback over-samples escalated/hard cases; the log is not the serving distribution. Owned by a uniform **random audit slice** (x% of traffic gets gold feedback regardless of confidence) — that slice, reported separately, is the honest accuracy. (Measured in the dogfood: biased feedback 0.72 vs audit 0.94 on the same system.) 2. **Eval burn** —