experiment-audit

A scientific reasoning engine for ML experiments.

Feed it claims and evidence — it checks for missing support, scopes evidence to claims, catches contradictions, scores confidence, and renders a structured scientific report. The kind of review a careful advisor would give your results before you write them up.

Created and maintained by Sree Dharshan G J

Most experiment-tracking tools show you numbers. experiment-audit checks whether your claim about those numbers actually holds up — missing evidence, out-of-scope comparisons, contradictions with earlier results, and confidence that isn't just assumed.

Install

pip install experiment-audit

Requires Python 3.11+. See Quick start below, or jump straight to Claude Code integration.

Quick start (30 seconds)

from experiment_audit.reasoning import (
    ScientificReasoningPipeline, ScientificReport,
    Claim, ClaimCategory, Scope,
)

claim = Claim(
    id="c1",
    subject="model-x",
    statement="model-x achieves 95% accuracy on CIFAR-10",
    category=ClaimCategory.PERFORMANCE,
    scope=Scope(dataset="cifar-10"),
)

pipeline = ScientificReasoningPipeline()
context = pipeline.build_initial_context(claims=[claim], evidence=[])
report = ScientificReport.from_pipeline_report(pipeline.execute(context))

print(report.to_markdown())

Claim → Evidence → Reasoning → Scientific Report. Every finding in the output traces back to specific evidence — the engine doesn't assert anything it can't point to.

Prefer the command line?

experiment-audit reasoning schema > claims.json   # see the expected input shape
experiment-audit reasoning run --input claims.json --format markdown

Claude Code Compatible

The reasoning discipline behind this project — how to phrase findings, weigh contradictory evidence, and write structured reviewer-style feedback — ships as a Claude Code skill, with the eight MCP audit tools available automatically wherever WANDB_API_KEY is set.

◆ Claude Code — run from inside this repo

/plugin marketplace add ./dev/experiment-audit-plugin
/plugin install experiment-audit@experiment-audit

It triggers automatically on prompts like:

• "Is this ablation confounded?"
• "Why did my loss crash?"
• "Review this paper's results claim."
• "Compare these ablation studies."
• "Write reviewer feedback on this."

See Quick start: the MCP server below for manual MCP setup, or the plugin's own README for full details.

Why experiment-audit?

Traditional experiment trackers are good at one thing: displaying metrics. They will happily tell you a run's final accuracy, loss curve, or config diff. What none of them do is check whether the sentence you're about to write about those numbers is actually supported by them.

experiment-audit treats a result the way a careful reviewer would before publication:

• Is there evidence behind this claim at all, or is it an assumption that snuck in?
• Does the evidence actually match what's being claimed — same dataset, same protocol, same scope — or is it being stretched to cover more than it proves?
• Does anything else you've measured contradict it?
• Is the confidence in the write-up proportional to the evidence, or borrowed from how confident the result felt?

This matters most for reproducibility, ablations, and the kind of paper-writing claims that are easy to overstate under deadline pressure — the exact places research claim verification tends to break down silently.

Who is this for

ML engineers	sanity-check a result before it ships in a report or a PR description
AI researchers	catch confounded ablations and out-of-scope comparisons before submission
Graduate students	get reviewer-style feedback on a results section before your advisor does
Research labs	a shared, deterministic check for scientific claims across a team's experiments
Academic / open-source projects	structured, evidence-traced scientific reports instead of ad hoc write-ups

What the reasoning engine does

Given a set of claims ("model-x achieves 95% accuracy on CIFAR-10") and the evidence backing them (metrics, configs, logs, prior runs), the engine runs six rules in sequence and produces a ScientificReport:

#	Rule	What it checks
1	Missing evidence	Does this claim have any supporting evidence trace at all?
2	Scope	Does the evidence actually match the claim's stated scope (same dataset, same hardware, same evaluation protocol)?
3	Contradiction	Does any other claim or evidence item conflict with this one?
4	Confidence	A computed score, not a guess — based on evidence quality, quantity, contradictions found, and what's missing.
5	Judgment	A verdict (supported / partially supported / unsupported) with the reasoning behind it.
6	Recommendation	What to do about it — gather more evidence, narrow the claim's scope, retract it.

Every finding traces back to specific evidence. Nothing in the report is an unsupported assertion — that would rather defeat the point.

This is one of two reasoning pipelines in the package. The second, lower-level pipeline (ScientificReasoningEngine — Evidence → Observations → Hypotheses → Confidence → Judgment → Recommendation) is a more generic, extensible framework for injecting custom hypothesis and confidence logic. Most people should start with the six-rule pipeline above. See src/experiment_audit/reasoning/__init__.py for both.

Features

Reasoning engine (the core)

• Claim and evidence modeling (Claim, EvidenceItem, Scope) with structured categories
• Six-rule scientific reasoning pipeline, run end-to-end or rule-by-rule
• Contradiction detection across claims and evidence
• Confidence scoring driven by evidence quality/quantity, not a fixed heuristic
• Structured ScientificReport — Markdown, JSON, or plain text
• Zero network calls; runs entirely on data you provide

Interfaces around the engine

• Python API — experiment_audit.reasoning, for embedding the pipeline in your own tooling
• CLI — experiment-audit reasoning run|schema
• Claude Code skill — the reasoning discipline as an installable skill, with worked examples
• MCP server — eight tools for auditing Weights & Biases runs directly from an agent
• Weights & Biases backend — read-only run/sweep/metric access behind the MCP tools

Quick start: the MCP server (W&B audit tools)

The original W&B experiment-audit tools are still here, unchanged, as an MCP integration. Set a read-only W&B API key:

export WANDB_API_KEY="your-read-only-key"
export WANDB_ENTITY="your-team-or-username"   # optional

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "experiment-audit": {
      "command": "experiment-audit-mcp",
      "env": {
        "WANDB_API_KEY": "your-read-only-key"
      }
    }
  }
}

Claude Code:

◆ Claude Code

claude mcp add -e WANDB_API_KEY=your-read-only-key experiment-audit -- experiment-audit-mcp

-e must come before the server name, not after — putting it after the name has been a source of "Invalid environment variable format" errors in some Claude Code versions.

Then ask your agent something like:

"Did I mess up my memory-ablation run? Compare mamfac-baseline and mamfac-no-memory in the mamfac project and check whether the only real difference is use_memory."

The agent calls audit_ablation, which returns a verdict (clean / confounded / uncertain), a confidence level, and the full config diff it based that verdict on.

Full tool reference (all eight tools, exact schemas, methodology) is in docs/design-spec-v1.md and docs/audit-methods.md — unchanged from the v1.0.0 release.

Architecture

experiment_audit/
├── reasoning/                 # the Scientific Reasoning Engine
│   ├── claims.py               # Claim, ClaimSet, Scope
│   ├── evidence.py              # Evidence, EvidenceItem (shared by both pipelines)
│   ├── contradictions.py        # Contradiction, ContradictionSet
│   ├── scientific_rules/        # the six concrete rules
│   │   ├── missing_evidence_rule.py
│   │   ├── scope_rule.py
│   │   ├── contradiction_rule.py
│   │   ├── confidence_rule.py
│   │   ├── judgment_rule.py
│   │   └── recommendation_rule.py
│   ├── rules.py                 # RuleContext, ScientificRule base
│   ├── pipeline.py              # ScientificReasoningPipeline: runs the six rules in order
│   ├── scientific_report.py     # ScientificReport: to_markdown/to_json/to_text
│   ├── observations.py          # generic pipeline: pattern detection over Evidence
│   ├── hypotheses.py             # generic pipeline: candidate explanations
│   ├── confidence.py             # generic pipeline: confidence scoring
│   ├── judgment.py                # generic pipeline: verdict rendering
│   ├── recommendation.py          # generic pipeline: recommendations
│   └── engine.py                  # ScientificReasoningEngine: the generic pipeline's orchestrator
├── cli.py                     # `experiment-audit reasoning run|schema`
├── models.py                  # RunRef, Run, MetricPoint, MetricHistory, Sweep, Page[T]
├── errors.py                  # ToolError + the frozen error_type taxonomy
├── server.py                  # FastMCP entrypoint; registers the 8 W&B audit tools
├── backends/
│   ├── base.py                # ExperimentBackend ABC, BackendCapability
│   ├── fake_backend.py        # in-memory test double
│   └── wandb_backend.py       # real W&B implementation
└── analysis/                  # the W&B audit tools' pure heuristics
    ├── comparison.py
    ├── divergence.py
    ├── confound.py
    └── sensitivity.py

The reasoning engine and the MCP/W&B layer are independent — the reasoning engine takes Claims and EvidenceItems directly and has no dependency on W&B, FastMCP, or any backend. Feeding W&B run data into the reasoning engine as claims/evidence (rather than hand-constructing them, as the quick-start example above does) is on the roadmap.

For the reasoning engine's design rationale, see research/07_reasoning_engine/ (reasoning-engine.md, reasoning-rules.md, confidence-system.md, evidence-model.md, scientific-reviewer.md). For the MCP/W&B layer's frozen contract, see docs/design-spec-v1.md.

Data handling

• Data never leaves your machine except calls to your own W&B endpoint (MCP layer only — the reasoning engine itself makes no network calls at all).
• Credentials are read once from environment variables, validated fail-fast on server