/tune-decide

name: tune-decide
description: The tunelab front door — decides whether a task needs fine-tuning at all, by running EXPERIMENTS on the user's data, not just interviewing. Use whenever the user wants to fine-tune, distill, or train a small/local model, cut their LLM API bill, replace frontier calls with something cheaper or faster, build a router/classifier/triage/cascade, asks "is fine-tuning worth it?" or "which architecture?", or wants to learn fine-tuning by experiment. Runs cheap probes + a frontier ceiling probe (headroom = ceiling − floor) and recommends an architecture with evidence. Even if the user has already decided to fine-tune, run this first. Routes to tune-data → tune-train → tune-eval for Levels 2–3; executes Levels -1/0/1 inline. Any hardware — NVIDIA/Linux users start here too; decide/data/eval are backend-agnostic, only the training step is MLX/Apple-Silicon.

tune-decide — should you fine-tune at all?

Most fine-tuning requests are better served by something cheaper. Your job: find the *lowest* level on the ladder that meets the user's bar, prove it with a runnable artifact when you can, and escalate only when the task demands it. Talking a user out of fine-tuning — by demonstrating a cheaper level meets their bar — is the success outcome and the trust engine of the whole product.

Step 0 — Read the project state before asking anything

On invocation, BEFORE asking the user a single question, check the project workdir:

• **`EXPERIMENT-LOG.md`** — prior interview answers, level decisions, runs, pre-registered bars. If a decision entry already exists, confirm it still holds instead of re-interviewing.
• **`runs/*/state.json`** — in-flight or interrupted training. If any has `"status": "running"` or `"interrupted"`, surface it immediately and offer to hand off to `tune-train` to re-attach (it re-derives health from the log tail). Schema (tune-train owns writing it; every skill may read it):

{ "run_id", "status": "running|interrupted|completed|failed", "pid", "command",
  "model", "adapter_path", "data_dir", "log_path", "total_iters", "save_every",
  "hparams": {"batch_size", "learning_rate", "num_layers", "max_seq_length"},
  "started_at", "updated_at", "best_val": {"iter", "loss"}, "resume_history": [] }

Training runs detached (`nohup <cmd> > runs/<id>/train.log 2>&1`, PID recorded); monitoring is polling the log file tail — never hold the training process in conversation context. Resume is weights-only in mlx-lm 0.31.3 (`--resume-adapter-file` restores weights, not optimizer state or the iter counter): completed iters = highest `NNNNNNN_adapters.safetensors` in `adapter_path`; rerun with `--iters <total minus completed>` + that checkpoint; expect a brief loss bump from cold optimizer state. A fresh session — or one that just compacted — resumes mid-pipeline from disk alone. Report what you actually found ("no EXPERIMENT-LOG.md in `<path>`"), and never assert a check you didn't run.

The capability ladder (first match wins, walking down)

| Level | Approach | Needs | Build time | When it wins | |---|---|---|---|---| | **-1** | Better prompt / cheaper API tier / prompt caching | nothing | minutes | Low volume (<1k calls/day), task still changing shape | | **0** | Embedding centroids — no training | ~10–20 examples/class | <1 hour | Crisp, well-separated buckets; semantic cache; router cold-start | | **1** | Embeddings + classifier (LR/XGBoost) | 200+ labels (LLM logs count) | <1 day | Fixed buckets, fuzzy boundaries, high volume — routers, gates, triage | | **2** | LoRA SFT on a 1–8B model (local, MLX) | 500–10k pairs | 1–3 days | Structured outputs, style transfer, narrow generation | | **3** | Continued pretraining + SFT (+ RAG hybrid) | ~10M+ domain tokens (relaxed in research mode) | weeks | Domain *fluency* the base model lacks; latency/offline motives |

Escapes that are not levels:

• **Knowledge tasks → RAG first.** "Make a model that knows our docs/API/policies" is retrieval, not training — fine-tuning teaches behavior and style, not reliable facts; a tuned model still hallucinates the details it was tuned on (see concepts/cpt-vs-rag.md — bundled at the plugin root, `../../concepts/` relative to this file). The Level-3 production pattern is CPT for fluency + RAG for fresh facts with citations.
• **Open-ended reasoning → stay on the frontier model**, with Level -1 optimizations. Distillation transfers narrow behavior, not general reasoning (see concepts/distillation.md).
• **Confidence routing everywhere.** Whatever ships, low-confidence inputs route to the frontier model. The hybrid beats either alone, and routed cases are the next training data.

Step 1 — Batch interview

One message, not twenty questions. Ask only what Step 0 and context didn't already answer:

1. **Task shape** — what goes in, what comes out, plus 2–3 *real* input/output examples. Fixed label set, structured object, or open text? 2. **Volume & economics** — calls/day, current model × tokens × cost, latency requirement. 3. **Data inventory** — logged LLM inputs/outputs? How many? Human-verified or raw? Any labeled data at all? Raw domain text (for CPT)? 4. **Motive** — cost, latency, privacy/on-device, offline, quality, or *understanding* (research mode — see below). Privacy/offline rules out Level -1 and forces a local level even at low volume. 5. **Hardware** — Apple Silicon (how much RAM), NVIDIA/Linux, or cloud-only.

**Research-mode recognition:** if the motive is understanding ("I want to *see* overfitting", "learn how LoRA works"), do not impose the production pipeline — no bar negotiation, no test-set ceremony. Route to a predict-then-run experiment (Step 5).

**NVIDIA/Linux users:** be honest — tune-train is MLX/Apple-Silicon today. decide, data, and eval are backend-agnostic by construction: run the decision here (a classifier needs no GPU at all), prepare standard JSONL with tune-data, train with TRL/Unsloth/axolotl or a cloud job, then return to tun