/tune-train

name: tune-train
description: Drive a local MLX-LM training run on Apple Silicon (LoRA/QLoRA, full fine-tuning, CPT) after tune-decide has validated a Level 2-3 plan. Use to pick a base model and hyperparameters, launch/monitor/resume a detached mlx_lm.lora run, diagnose loss curves, run continued pretraining on a validated corpus, or fuse adapters / export GGUF. Also the re-entry point when a training run was interrupted or a session died mid-run. Assumes tune-decide already validated the level — routes there first if no decision is on disk.

tune-train — local training on Apple Silicon (MLX-LM)

Drives `mlx_lm.lora` over a validated `data/` directory from tune-data (`train.jsonl`/`valid.jsonl`/`test.jsonl`). Full verified CLI reference: `references/mlx-reference.md` (mlx-lm 0.31.3). `<skill-dir>` below = the directory containing this SKILL.md; run commands from the user's project workdir.

**Teaching default (explain-why protocol):** every step you run gets four short lines before — **What** we're doing · **Why** (the failure it prevents) · **Expect** (healthy output) · **Read** (how to interpret what came out) — and one line after connecting result → next decision. One-liners, not essays. Define jargon inline on first use, pointing at the bundled concepts files for depth (plugin root, `../../concepts/` relative to this file). If the user says "skip the teaching" (or is clearly expert): drop Why/Expect/Read, keep What + the result reading.

**Stop-and-ask points** (pre-registration; these exactly, nowhere else): the level recommendation (tune-decide), the labeling prompt (tune-data), the acceptance bar AND metric set (registered by tune-decide at decision time; must be on disk before any training launch), and any expensive run — which here means every training launch (Step 4).

Step 0 — Read the project state from disk FIRST

Before asking the user anything:

1. **Read `EXPERIMENT-LOG.md`** in the workdir. tune-decide wrote the interview summary and level decision there; tune-data wrote data provenance. Never re-ask what's already answered. **No level decision for this task → do not train; route to tune-decide first.** tune-train assumes a validated Level 2/3 decision — for fixed-label outputs especially, a Level-1 classifier usually makes this whole skill unnecessary. 2. **Scan `runs/*/state.json`.** For any run with `"status": "running"`: is the PID alive (`ps -p <pid>`)? Is the log tail fresh (`tail -n 30 <log_path>`, recent mtime)? Alive + fresh → offer to re-attach and go straight to Step 5 monitoring. Dead with iters remaining → set `"status": "interrupted"` and offer the Step 6 resume.

A fresh session — or one that just compacted — must be able to pick up mid-pipeline from `EXPERIMENT-LOG.md` + `state.json` + `train.log` alone. Append every decision this skill makes to `EXPERIMENT-LOG.md` as `## <date> — <event>` with short `Decision:` / `Run:` / `Result:` / `Predicted-vs-actual:` / `Lesson:` lines as applicable, each with rationale.

Step 1 — Preflight

python3 -c "import platform; assert platform.machine() == 'arm64', 'Apple Silicon required'"
uv tool install mlx-lm        # installs the mlx_lm.* commands (verified 0.31.3)
sysctl -n hw.memsize          # bytes → RAM ceiling below

| RAM | Ceiling | |---|---| | 8 GB | ~3B at 4-bit (batch 2, expect `--grad-checkpoint` at 4B) | | 16 GB | 8B at 4-bit is the ceiling — Qwen3-8B-4bit is 4.6 GB of weights + 2–4 GB training overhead; use `--grad-checkpoint` | | 32 GB+ | 8B comfortable; 4-bit 14B possible with `--grad-checkpoint` |

Not on Apple Silicon (e.g. NVIDIA)? Be honest: this training backend is MLX-only today. tune-decide/tune-data/tune-eval are backend-agnostic — the JSONL chat data feeds TRL/Unsloth/axolotl or a cloud job directly; train there, then return to tune-eval for the scoreboard.

Step 2 — Pick the base model

Smallest plausibly-capable wins: distillation transfers narrow behavior, and smaller = faster training, faster inference, easier deployment. Verified table (2026-06, all ids returned HTTP 200):

| Task | Start | Escalate | |---|---|---| | classification / routing / extraction | `mlx-community/Qwen3.5-0.8B-MLX-4bit` (0.63 GB) | `mlx-community/Qwen3.5-2B-4bit` (1.72 GB) | | structured output / JSON | `mlx-community/Qwen3-4B-Instruct-2507-4bit` (2.26 GB) | `mlx-community/Qwen3.5-4B-4bit` (3.03 GB) | | style-transfer / prose generation | `mlx-community/gemma-3-4b-it-qat-4bit` (3.00 GB) | `mlx-community/Qwen3-8B-4bit` (4.61 GB) | | CPT (base, non-instruct) | `mlx-community/Qwen3-0.6B-Base-4bit` (0.34 GB) | `mlx-community/SmolLM3-3B-Base-4bit` (1.73 GB) |

• **Why 2507 for JSON:** it ships a *non-thinking* chat template — output starts at your schema's first byte. Qwen3 (non-2507) and Qwen3.5 templates are hybrid-thinking and emit `<think>` blocks by default: raw `mlx_lm.generate` output will show them (normal, not damage); tune-eval's `run_test_set.py` disables and strips them.
• **CPT warning:** most instruct checkpoints have NO base 4-bit twin — Qwen3 1.7B/4B/8B-Base-4bit and *all* Qwen3.5-Base-4bit don't exist on mlx-community. Base 4-bit under ~3.5 GB: Qwen3-0.6B-Base, LFM2.5-1.2B-Base, SmolLM3-3B-Base, gemma-3-{1b,4b}-pt.
• **Verify before ANY download** — it is one curl; repo names churn, and a dead multi-GB pull is just the expensive version:

curl -s -o /dev/null -w '%{http_code}' https://huggingface.co/api/models/mlx-community/Qwen3.5-2B-4bit
# 200 = exists. 401 = missing or gated — the HF API returns 401, NOT 404, when unauthenticated.

Suffix traps: `-MLX-4bit` and `-4bit` are duplicate uploads (same weights); `OptiQ` is mixed-precision and much larger — never substitute it for plain 4-bit.

**QLoRA, framed once:** LoRA trains small fp16 adapter matrices on a frozen base (the base model's billions of parameters never change — see `../../concepts/lora-vs-qlora.md`). Point `mlx_lm.lora` at a 4-bit checkpoint and you get **QLoRA automatically** — fp16 adapters over