/tune-data

name: tune-data
description: Build a training dataset for fine-tuning, distillation, or continued pretraining. Use when the user wants to turn logs/CSV/JSONL into fine-tuning data, label data with an LLM, distill a teacher model's outputs, generate synthetic training examples from nothing, chunk raw domain text for CPT, deduplicate a dataset, make train/valid/test splits, convert to MLX chat format, write a datacard, or asks "how much data do I need to fine-tune?".

tune-data — build the dataset

Data quality determines fine-tuning quality more than any hyperparameter. The pipeline: **ingest → (distill) → dedupe → split → validate → datacard**. Every step has a bundled script; chain them, don't skip the gates.

`<skill-dir>` below = the directory containing this SKILL.md. Stdlib scripts run with `python3`; `distill_generate.py` is PEP 723 (`uv run`). Run everything from the user's project workdir.

Before asking the user anything: read the project state

On invocation, check the workdir **first** — a fresh session (or one that just compacted) must resume mid-pipeline from disk alone:

1. **`EXPERIMENT-LOG.md`** — tune-decide writes the interview summary and level decision here precisely so you never re-ask. Look for: task shape, data inventory, the chosen level, any frozen labeling prompt or dedupe threshold from a prior session. 2. **`runs/*/state.json`** — if any run has `status: running|interrupted`, training is using `data_dir` right now (or will resume into it). Do not regenerate splits underneath it; ask before touching that directory. 3. **Partial pipeline artifacts** — resume where disk says you are: a raw teacher-output file smaller than the input means resume labeling (the script skips done ids; session-native, count ids and continue); `deduped.jsonl` present means go to split; `data/{train,valid,test}.jsonl` present means re-run validate and go to the datacard.

If there is **no level decision** in EXPERIMENT-LOG.md, route to **tune-decide** before building anything — whatever the task shape. Classification smell (N fixed categories, labels already logged) is the most urgent case: a Level 1 embeddings+classifier may need no fine-tuning dataset at all, and proving that in 10 minutes beats preparing data for a LoRA the user doesn't need.

After every completed stage, append to `EXPERIMENT-LOG.md` (append-only, `## <date> — <event>` with short Decision / Run (config) / Result / Predicted-vs-actual / Lesson lines as applicable). That log is what makes the dataset reproducible.

Teaching protocol

Every step below is framed as four short lines before running it — **What** we're doing · **Why** it matters (the failure it prevents) · **Expect** what healthy output looks like · **Read** how to interpret what came out — and one line after connecting result → next decision. One-liners, not essays; jargon defined inline on first use, with depth in the bundled concepts files (`../../concepts/` relative to this file, e.g. concepts/epochs-and-overfitting.md). If the user says "skip the teaching" (or is clearly expert): drop Why/Expect/Read, keep What + the result reading.

**Stop-and-ask checkpoints.** tune-data stops for user judgment at exactly two points, nowhere else: **the prompt-freeze checkpoint** — freezing the labeling/teacher prompt after the 25-sample spot-check — and **the expensive-run checkpoint** — before a full API labeling job or anything that costs real money or hours. The level recommendation was tune-decide's checkpoint (read it from the log); the acceptance bar and metric set are registered by tune-decide at decision time and confirmed in tune-eval before any scoring.

**Research mode.** If EXPERIMENT-LOG.md marks the project research mode (or the goal is understanding, not shipping): build the smallest dataset that serves the experiment. Keep the train/valid split — val loss is the instrument. Skip the datacard and test ceremony. For the dedupe-ablation experiment, planted duplicates are the point: dedupe one arm only, never both.

Target formats (MLX-LM, auto-detected)

Output is a directory (conventionally `data/`) with `train.jsonl` / `valid.jsonl` / `test.jsonl`, one JSON object per line, all in ONE of:

| Format | Line shape | Use for | |---|---|---| | **chat** | `{"messages": [{"role": "system"...}, {"role": "user"...}, {"role": "assistant"...}]}` | SFT — the default. System turn optional; keep it identical across records if used | | **tools** | chat messages (assistant turns may carry `tool_calls`) + top-level `"tools": [...]` | tool-call SFT | | **completions** | `{"prompt": "...", "completion": "..."}` | SFT without chat structure | | **text** | `{"text": "..."}` | continued pretraining (CPT) on raw domain text |

These are standard JSONL — any trainer consumes them (TRL/Unsloth/axolotl on NVIDIA, cloud jobs). tune-data is backend-agnostic by construction; only the training step is MLX-first.

Step 1 — pick the path by what exists

| Path | You have | Route | |---|---|---| | **A** | labeled/paired data (logged LLM calls, human-labeled CSV) | convert to MLX format → Step 2 | | **B** | unlabeled inputs, teacher must label | teacher tier below → Step 2 | | **C** | nothing | synthetic inputs + teacher labels → Step 2 | | **D** | raw domain text (docs, filings, code) for CPT | chunk → Step 2 |

Sizing, when the user asks "is this enough?":

| Task | Minimum to try | Comfortable | |---|---|---| | Classification SFT | 50–100/class | 500+/class | | Generation/extraction SFT | 500 pairs | 1k–10k | | CPT | under ~10M tokens, question whether CPT is worth it at all | ~10M+ tokens |

Always: 1,000 clean, deduped, diverse examples beat 10,000 noisy ones.

Path A — labeled data exists

Conversion is task-specific — write a small throwaway script. Keep the original input text and any **stable id** in each record (ids survive into resume logic and the datacard). Map: input → user turn, logged output/label → assistant turn, one shared system prompt de