/ml-verify

name: ml-verify
description: Use when the user wants to verify code, config, or math before running — or proactively before any expensive training job or deployment

ML Verification

Catch mistakes before they waste GPU hours. Verify configs, code, and math against documented framework behavior.

Grounding

**Detect mode:** On your first grounding call, check if Leeroopedia KB tools are available. If they return results, use **KB mode**. If unavailable or auth fails, use **Web mode**.

**KB mode:** Call `verify_code_math` / `query_hyperparameter_priors` / `review_plan`. Cite as `[PageID]`.

**Web mode:** WebFetch API docs for every non-trivial import, verify signatures and params against official docs, WebFetch known good configs for comparison. Cite as `[DocName: specific page/section](URL#anchor)` — never use generic `[source]`. The link text MUST name the document and section (e.g., `[HF PEFT: LoRA Conceptual Guide](https://huggingface.co/docs/peft/main/en/conceptual_guides/lora)`). Start response with: `> Grounding: Web mode — citations from official docs.`

**Web mode URL registry:**

• HF PEFT LoRA guide: `https://huggingface.co/docs/peft/main/en/conceptual_guides/lora`
• HF PEFT quickstart: `https://huggingface.co/docs/peft/main/en/quicktour`
• HF Transformers TrainingArguments: `https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments`
• HF TRL SFTTrainer: `https://huggingface.co/docs/trl/main/en/sft_trainer`
• HF TRL SFTConfig: `https://huggingface.co/docs/trl/main/en/sft_trainer#trl.SFTConfig`
• DeepSpeed: `https://www.deepspeed.ai/docs/config-json`
• vLLM: `https://docs.vllm.ai`
• PyTorch: `https://pytorch.org/docs/stable`

The Iron Law

NO TRAINING RUN WITHOUT VERIFICATION FIRST

An hour of verification saves days of debugging failed runs. Check the config against KB-documented ranges, check the code against documented API contracts.

Phases

Phase 1: Check Against Documentation

**KB mode:**

Call the appropriate KB tools:

• **For code/math:** `verify_code_math(code_snippet, concept_name)`
• **For configs/hyperparameters:** `query_hyperparameter_priors(query)` with model size, task type, hardware, and framework context
• **For full training configs:** `review_plan(proposal, goal)` with the complete config

Run whichever combination fits. When in doubt, run all applicable checks in parallel. Cite as `[PageID]`.

**Web mode:**

WebFetch the relevant documentation for each check:

• **For code/math:** WebFetch the API docs for the framework. Verify function signatures, parameter names, and return types against the official docs.
• **For configs/hyperparameters:** WebFetch the framework's config reference page and known-good example configs (e.g., Axolotl examples, HF training examples). Compare user values against documented defaults and recommendations.
• **For full training configs:** WebFetch docs for each major config section (model, optimizer, data, distributed). Cross-check all values.

Cite as `[DocName: section](URL#anchor)` — never generic `[source]`. Start response with: `> Grounding: Web mode — citations from official docs.`

**Web mode hard gate:** You MUST call WebFetch on at least 3 URLs from the registry BEFORE writing ANY findings table row. Extract exact parameter defaults, API signatures, and recommended ranges from fetched content. Constructed URLs that were never fetched score 0 on grounding — the judge checks for actual page fetch evidence. Do NOT cite a URL you did not fetch.

**Extract-and-quote rule:** After each WebFetch, write down 2-3 exact values from the page (default LR, parameter type, version-specific behavior) as scratch notes. When writing findings rows, QUOTE these extracted values — e.g., "TRL 0.12 defaults SFTConfig.learning_rate to 2e-5" not just "typical LR is 2e-4". If the fetched page shows a value different from your prior belief, surface the discrepancy explicitly: "Note: docs say X, common advice says Y — using doc value."

**Version-specificity gate:** Every web-mode citation MUST include the framework version or doc date when available. E.g., `[HF PEFT v0.13: LoRA Conceptual Guide](URL)`. If the fetched page shows a version number, include it. If not, append `(undated)` to the citation.

**If BOTH KB and web are unavailable:** 1. First line: `⚠️ WARNING: This verification is ungrounded. All recommendations below are best-effort. Verify independently.` 2. Every row in the findings table ends with `**UNGROUNDED**` 3. Cite specific public sources where possible: arXiv IDs, doc URLs, framework doc sections

**Specificity rule**: Never recommend a range when you can recommend a value. Pick the single best value from your sources and cite why.

**Gate**: Every parameter and code path has been checked against documentation. If any check couldn't be verified, flag it in the findings table.

Phase 2: Dry Run Checklist

Before the real run, verify these can complete without error:

• [ ] Model loads on target hardware (no OOM on init)
• [ ] Data pipeline produces correctly shaped batches
• [ ] Forward + backward pass completes (1 step, no crash)
• [ ] Loss is a reasonable initial value (cross-entropy on vocab: expect `ln(vocab_size)` ≈ 10-11 for 32k vocab; flag if <1.0 or >15.0 or NaN)
• [ ] Gradient norms are in expected range (0.1–10.0 for LLM fine-tuning; flag if >100 or exactly 0.0)
• [ ] Checkpoint save/load works
• [ ] Estimate total VRAM with exact formula:
• [ ] Verify every FAIL/WARN fix is copy-paste ready (run the Verify command from Issues Found)
• [ ] Cross-check every numerical claim in findings table against a fetched doc value — if you wrote "typical range is X" but docs say Y, fix or flag the discrepancy
• QLoRA 4-bit: `(params × 0.5B) + (trainable_params × 2B × 3 for AdamW) + (batch × seq_len × hidden × n_layers × 2B for activations)`
• Full FT bf16: `(params × 2B) + (params × 2B × 3 for AdamW) + activations`
• Flag if >85% of GPU RAM. Show the