/experiment-audit

name: experiment-audit
description: Use this skill for scientific and ML-research reasoning work — evaluating experimental claims, auditing training runs or ablations, checking whether a statistical claim holds up, assessing reproducibility, reconciling contradictory results, reviewing a paper's methodology or results section, writing reviewer-style feedback, or producing a structured research report. This is a scientific reasoning discipline, not just a tool wrapper, so it applies even with no live data source — e.g. reviewing a pasted table of results, sanity-checking a claimed effect size, or evaluating an ablation described in prose. Trigger on phrasing like "did I mess up this experiment," "is this result real," "why did my loss/reward do X," "which run is better," "is this ablation confounded," "review this paper's claims," "write reviewer feedback," "is this reproducible," "what should I conclude from this," or "write up these results." When the user's data lives in Weights & Biases, this skill also covers the experiment-audit-mcp integration's eight tools (test_connection, list_runs, get_run_summary, get_metric_history, compare_runs, audit_training_curve, audit_ablation, audit_sweep) for pulling that evidence in — but the reasoning discipline in this skill is the primary thing, and applies whether or not those tools are called.

Experiment Audit — Scientific Research Reasoning Engine

What this is

Experiment Audit is a **scientific research reasoning engine**: a discipline for evaluating experimental and empirical claims the way a careful reviewer or co-author would — checking what the evidence actually supports, separating measurement from interpretation, naming uncertainty precisely, and never letting a clean-looking number stand in for a checked one.

The **MCP server** (`experiment-audit-mcp`, eight tools over a user's Weights & Biases project), the **CLI**, and the **Python package** are integrations of this engine — they are how it pulls structured evidence out of a live experiment-tracking backend when one is available. They are not what this skill is *for*. A huge share of real requests in this domain — reviewing a paper, sanity-checking a table someone pasted in, writing reviewer feedback, reasoning about an ablation described in prose — involve no MCP call at all, and the same reasoning discipline applies to all of them.

Think of it as two layers:

1. **The reasoning engine** (this skill, `prompts.md`, `examples.md`) — how to evaluate evidence, phrase findings, hedge accurately, catch contradictions, and write up conclusions, regardless of where the evidence came from. 2. **The MCP integration** (`reference.md`, the tool table below) — the specific, calibrated tools available when the evidence lives in W&B: what each one computes, its exact thresholds, and its documented blind spots.

Layer 1 is always active when you're reasoning about an experiment. Layer 2 activates only when there's a live W&B project to query.

When to use this skill

• Diagnosing a training curve, run, ablation, or sweep — whether the

data comes from a live W&B project or numbers/plots the user pasted.

• Evaluating whether an experimental or statistical claim is supported

by the evidence given (a paper's results table, a reported p-value, an effect size, a reproducibility claim).

• Reviewing a paper's methodology or results section, or drafting

reviewer-style feedback on one.

• Reconciling two results, runs, or papers that appear to disagree.
• Writing up findings as a structured research report, audit summary,

or reviewer comment.

• Sanity-checking W&B credentials, browsing runs/sweeps, or pulling

metric history via the MCP integration.

When NOT to use this skill

• **General ML/coding help that isn't evaluating a claim or a result**

— writing training code, debugging a stack trace, explaining what an architecture does in the abstract. Use normal coding assistance; pull in this skill only once there's a result or a claim to reason about.

• **Requests with no experimental content** — this skill is specific to

empirical/experimental reasoning, not general research-writing help (e.g. "help me write a related-work paragraph" on its own doesn't need it, unless it also involves evaluating what those related papers actually showed).

• **When there isn't enough information to reason from and the user

hasn't asked for a read anyway.** If someone asks "is my model good" with no metrics, curve, or claim attached, ask what evidence they have rather than inventing plausible-sounding numbers to audit.

• **Don't reach for an MCP `audit_*` call, or its heavier judgment

language, when the user only wants raw data or a plain diff** — see "Choosing between the reasoning engine and MCP tools" below.

Choosing between the reasoning engine and MCP tools

Three situations, three different moves:

1. **The user references a live W&B project, run, or sweep** (names an entity/project, a run ID, a sweep ID, or says something like "my project" / "my runs"). → Use the MCP integration. Pick the specific tool with the selection guide below; read `reference.md` for exact signatures before calling anything you're unsure of.

2. **The user hands you data directly** — a pasted metrics table, a screenshot's numbers, a paper's reported statistics, a described ablation with baseline/variant numbers. → Reason directly using the principles in `prompts.md`. You are doing the same job an `audit_*` tool does (diagnose, don't just describe), but by hand — so **hold yourself to the same discipline**: name your method explicitly (e.g. "a two-sample comparison assuming roughly normal errors," "eyeballing the curve for a discontinuity"), state your own confidence, and be explicit that this is your informal read, not a calibrated tool's output, since the two carry different evidentiary weight for the user. Never present your