/tune-eval

name: tune-eval
description: Evaluate a fine-tuned, distilled, or continued-pretrained model with held-out test discipline — the honest scoreboard at the end of the tunelab pipeline. Pre-registers the acceptance bar and metric set BEFORE results exist, runs the untouched test split through base and tuned models, scores classification (accuracy, macro-F1, per-class precision/recall, confusion matrix, hallucinated-label flagging) or generative output (blinded pairwise LLM-as-judge — session-native or API), measures CPT perplexity deltas and catastrophic forgetting, and drives the ship / more-data / debug / escalate decision plus drift monitoring. Use whenever the user asks "is my fine-tuned model actually good?", wants to compare tuned vs base vs teacher, needs an eval methodology or wants to score against a held-out test set, asks about win rates, F1, confusion matrices, judge bias, or perplexity, wants to re-score a deployed model for drift, or asks about validation vs test discipline.

tune-eval — the honest scoreboard

Evaluation answers one question: **does the tuned model meet the pre-registered bar on data it has never influenced?** The validation set already steered training; only `test.jsonl` — untouched until now — gives an honest number (see concepts/validation-vs-test.md — bundled at the plugin root, `../../concepts/` relative to this file).

Teaching note: each step below is framed as four short lines before running it — **What** we're doing · **Why** (the failure it prevents) · **Expect** (healthy output) · **Read** (how to interpret what came out) — and one line after connecting result → next decision. One-liners, not essays; define jargon inline on first use with a concepts/ pointer. If the user says "skip the teaching" (or is clearly expert), drop Why/Expect/Read and keep What plus the result reading.

`<skill-dir>` below = the directory containing this SKILL.md; run commands from the user's project workdir.

Step 0 — Read the project state from disk (before asking anything)

On invocation, BEFORE asking the user a single question, check the project workdir:

1. **`EXPERIMENT-LOG.md`** — prior decisions, the tune-decide interview summary, training runs, and (critically) whether a bar + metric set was already pre-registered. tune-decide writes the interview and level decision there precisely so later skills — and later sessions — never re-ask. If a bar exists, confirm it in one line; do not renegotiate. Also check whether the current `test.jsonl` was already spent by a previous eval round. 2. **`runs/*/state.json`** — the run-continuity contract (tune-train owns writing it; all skills may read it):

{ "run_id", "status": "running|interrupted|completed|failed", "pid", "command",
  "model", "adapter_path", "data_dir", "log_path", "total_iters", "save_every",
  "hparams": {"batch_size", "learning_rate", "num_layers", "max_seq_length"},
  "started_at", "updated_at", "best_val": {"iter", "loss"}, "resume_history": [] }

This file hands you `model`, `adapter_path`, and `data_dir` — build every command below from it instead of asking. If `status` is `running`, do not eval a moving target: poll the log tail (`tail runs/<id>/train.log`), never hold the training process in conversation context. If `interrupted`, route back to tune-train first — resume is weights-only in mlx-lm 0.31.3 (`--resume-adapter-file` restores weights; fresh optimizer, iter counter resets): completed iters = highest `NNNNNNN_adapters.safetensors` in `adapter_path`; rerun with `--iters <total minus completed>` plus that checkpoint, and expect a brief loss bump from cold optimizer state. Only `completed` runs get the scoreboard.

A fresh session — or one that just compacted — must be able to resume mid-pipeline from these two artifacts alone.

Three rules that protect the answer

1. **The bar is set before results exist — including which metrics.** Pre-registration covers the metric card (from the family table below), the guardrails, and the number to beat. A bar chosen after seeing results is a rationalization with a decimal point. 2. **One look at test.** Run the test set once, report, decide. If the result triggers a retraining round, that test set is *spent* — the next model was chosen partly because of it, so future comparisons against it flatter you. Note the spend in EXPERIMENT-LOG.md and carve a fresh test split from new data next round. 3. **Compare three ways, not one.** Tuned vs **base** = did training do anything. Tuned vs **teacher** = what distillation lost (see concepts/distillation.md). Vs **gold labels** where they exist = absolute truth. In the distillation case the teacher's outputs *are* `expected`, so tuned-vs-teacher comes free.

**Research mode is exempt from the ceremony.** If the goal is understanding (overfit-on-purpose, rank sweeps, tiny CPT), there is no acceptance bar to negotiate — the "eval" is comparing the user's *prediction* against the actual curve, logged as Predicted-vs-actual in EXPERIMENT-LOG.md.

Step 1 — Pre-register the bar and metric card (stop and ask)

This is a hard checkpoint, normally already satisfied: the bar is registered by tune-decide at decision time and must exist before any training launch — if it is in the log, confirm it in one line and move on; do not renegotiate. Step 1 is the **fallback registration** for sessions entering the pipeline mid-way (e.g. evaluating an externally trained model): present the family's metric card, ask for the bar, and append both to EXPERIMENT-LOG.md **before any prediction run**. For generative tasks, the judge criteria are part of the card.

| Task family | Core metrics | Guardrails | |---|---|---| | Classification | accuracy, macro-F1, per-class P/R | cost-weighted confusion cells (user names the expensive ones), calibration, coverage-at-threshold for routing | | Generative (SFT) | judge win-rate vs base & vs teacher | format-validity rate (JSON parse %), faithfulness, length calibration | | E