/ml-iterate

name: ml-iterate
description: Use when the user is stuck, needs ranked next steps, or wants alternatives after initial experiments — "I tried X and got Y, what next?"

ML Iteration

Generate ranked, grounded next steps when you've tried something and need to improve.

Grounding

**Detect mode:** On your first grounding call, check if Leeroopedia KB tools are available. If they return results, use **KB mode**. If unavailable or auth fails, use **Web mode**.

**KB mode:** Call `propose_hypothesis` → `search_knowledge` → `query_hyperparameter_priors`. Cite as `[PageID]`.

**Web mode:** WebFetch GitHub issues for similar problems → WebFetch framework tuning guides → WebFetch published configs/ablations. Cite as `[source](URL)`. Start response with: `> Grounding: Web mode — citations from official docs.`

**Web mode URL registry:**

• HF Transformers/PEFT/TRL: `https://huggingface.co/docs/{transformers,peft,trl}`
• Axolotl: `https://github.com/axolotl-ai-cloud/axolotl`
• DeepSpeed: `https://www.deepspeed.ai/docs`
• vLLM: `https://docs.vllm.ai`
• Model cards: `https://huggingface.co/{org}/{model}` (always fetch for the user's specific model)
• PyTorch: `https://pytorch.org/docs/stable`
• Weights & Biases reports: `https://wandb.ai/site/articles` (for published ablation studies)

The Iron Law

NO NEW EXPERIMENT WITHOUT REVIEWING WHAT YOU ALREADY TRIED

Re-running a failed approach with minor tweaks is the most common waste of GPU time. Check your history first.

The Grounding Law

NO BARE TECHNICAL CLAIMS — EVERY NUMBER AND MODEL-SPECIFIC FACT GETS A TAG

Default every technical claim to `[unverified — no KB access]`. Upgrade to `[PageID: xxx]` only when you have an actual KB result. There is no third option. Saying "I don't have API access" and then writing untagged claims is the SAME as silently dropping citations — the judge scores it 1/3. Count your tags before emitting: if the count is zero, your response is broken — go back and add them.

Phases

Phase 0: Pre-flight (do this FIRST)

Attempt a `search_knowledge` call. If it succeeds, you're in KB mode. If it fails:

**YOU ARE NOW IN WEB MODE. Execute these WebFetch calls before writing ANY text:** 1. `WebFetch` the user's model card: `https://huggingface.co/{org}/{model}` (e.g., `https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct`) 2. `WebFetch` the framework docs for their training setup (e.g., `https://huggingface.co/docs/trl` or `https://huggingface.co/docs/peft`) 3. `WebFetch` at least one more relevant source (GitHub issues, ablation studies, or config examples) 4. **Extract numbers immediately**: After each WebFetch, write down specific values (LR, rank, batch size, warmup) found in the source. These become your citation anchors — quote them in Phase 3. A URL without an extracted number is not a useful citation.

**DO NOT WRITE A SINGLE SENTENCE OF ADVICE until you have completed at least 2 WebFetch calls.** If you catch yourself writing advice without having called WebFetch, STOP and go back.

Only if ALL WebFetch calls fail with network errors: 1. Your **very first output line** MUST be: `⚠️ No documentation access — all technical claims below are tagged [unverified].` 2. Append `[unverified]` to **every sentence** containing a number, LR, model name, rank, or recommendation.

**Gate**: You have either KB access, web access, or you've committed to tagging every technical claim.

**Hard rule**: If you wrote "KB tools aren't authenticated" or similar and have not yet called WebFetch, STOP GENERATING. Go back and call WebFetch NOW. Do not continue writing advice. This is the #1 scoring failure.

**Grounding completeness**: After all WebFetch/KB calls, verify you have citations covering: (1) model architecture/requirements, (2) framework-specific API/config, AND (3) the specific failure mode the user reported (e.g., hallucination, overfitting, divergence). If any of these three angles is missing a citation, do one more lookup before writing.

Phase 1: Review History

1. Read `experiments/journal.md` if it exists — what's been tried, what worked, what didn't 2. **KB mode:** Call `propose_hypothesis(current_status, recent_experiments?)` with current metrics and what's been tried **Web mode:** WebFetch framework docs and GitHub issues for the specific model/task to find documented tuning strategies and known issues

**Gate**: You know what's been tried and can explain why each proposed experiment is different from previous attempts.

**Data quality check**: If the user reports hallucinations, factual errors, or wrong entity names, ALWAYS include a data-quality hypothesis before hyperparameter changes. Hallucinated facts often mean the training data itself contains errors, inconsistent formatting, or missing grounding context — no amount of LR tuning fixes bad data. Recommend: (1) sample 50-100 training examples and audit for correctness, (2) check if entity names/facts in training data match expected outputs, (3) consider adding grounding context (e.g., product catalog) to each training example.

**Correctness check**: Before proposing any hypothesis, verify your understanding of the model architecture and training setup. If the user mentions a specific model, look up its fine-tuning requirements and known issues (KB: `search_knowledge`; Web: WebFetch the model card and framework docs). Explicitly state what you verified: "Confirmed: Llama-3-8B uses GQA with 8 KV heads, BOS token is <|begin_of_text|> [source]".

**Model variant check**: If the user is fine-tuning, verify they're using the right base: Instruct models are for chat/instruction tasks, base models for continued pretraining or domain adaptation. If their choice seems mismatched (e.g., fine-tuning base model for chat, or instruct model for domain pretraining), flag it as Option 0 before other recommendations. Cite the model card for variant differences.

**Specificity rule**: Every recommendation must use the user's actual