/experiment-audit

name: experiment-audit
description: "Audit experiment integrity before claiming results. Uses cross-model review (external reviewer backend) to check for fake ground truth, score normalization fraud, phantom results, and insufficient scope. Use when user says \"审计实验\", \"check experiment integrity\", \"audit results\", \"实验诚实度\", or after experiments complete before writing claims."
argument-hint: "[experiment-dir-or-results-path]"
allowed-tools: Bash(*), Read, Write, Edit, Grep, Glob, mcp__codex__codex, mcp__codex__codex-reply, mcp__manual_review__review, mcp__manual_review__review_reply, mcp__antigravity__antigravity, mcp__antigravity__antigravity-reply, mcp__grok__grok, mcp__grok__grok-reply, mcp__claude-review__review, mcp__claude-review__review_reply, mcp__claude-review__review_start, mcp__claude-review__review_reply_start, mcp__claude-review__review_status

Experiment Audit: Cross-Model Integrity Verification

> **Before each reviewer slot**, follow [Selectable Reviewers](../shared-references/reviewer-backends.md): > resolve invocation → `.aris/reviewer.json` → Claude Code default (Codex MCP) > with `reviewer_route.py` through the canonical helper resolver, preserving this > slot's tier, fresh contexts, source/image inputs and deterministic gates. > Codex examples/pins below apply only to `backend=codex`; other routes use their > own first-call/reply contract. Save the route and actual returned identity; > unavailable selections are `REVIEW_UNAVAILABLE`. Classify acceptance from actual > families/completion/scope. Forward invocation reviewer/model/effort overrides > to reviewer-bearing children. Generation workers are unchanged.

> 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is > verdict-bearing — it judges experiment integrity. Re-running that verdict on a > timer adds no new signal, and a loop that accepts its own output to decide > when to stop crosses into self-acquittal (`acceptance-gate.md`). Schedule the > *external wait that precedes it* — experiments done → then audit **once**. See > [`shared-references/external-cadence.md`](../shared-references/external-cadence.md).

Audit experiment integrity for: **$ARGUMENTS**

Why This Exists

LLM agents can produce fraudulent experimental results through: 1. **Fake ground truth** — creating synthetic "reference" from model outputs, then reporting high agreement as performance 2. **Score normalization** — dividing metrics by the model's own max to get 0.99+ 3. **Phantom results** — claiming numbers from files that don't exist or functions never called 4. **Insufficient scope** — reporting 2-scene pilots as "comprehensive evaluation"

These are NOT intentional deception — they are failure modes of optimizing agents that lack integrity constraints. This skill adds that constraint.

Core Principle

**The executor collects file paths. The external reviewer backend reads code and judges integrity. The executor does NOT participate in integrity judgment.**

This follows `shared-references/reviewer-independence.md` and `shared-references/experiment-integrity.md`.

Constants

• **REVIEWER_BACKEND = `codex`** — Default: Codex MCP (ultra). Override with `— reviewer: oracle-pro` for Oracle MCP, or `— reviewer: manual` for Manual Review MCP. If manual-review MCP is unavailable, stop and print the install command; do not fall back to Codex. See `shared-references/reviewer-routing.md`.

Reviewer Calling Convention

When calling the reviewer, branch on REVIEWER_BACKEND:

**If REVIEWER_BACKEND = `codex`:** Use `mcp__codex__codex` for new review threads. Use `mcp__codex__codex-reply` for follow-up rounds (reuse threadId).

**If REVIEWER_BACKEND = `manual`:** Use `mcp__manual_review__review` for new review threads with: prompt: [exact same prompt that would go to Codex] config: {"model_reasoning_effort": "xhigh", "executor_model": "<actual executor model>", "require_reviewer_model": true} Save the returned `threadId`. Use `mcp__manual_review__review_reply` for follow-up rounds with: threadId: [saved manual-review threadId] prompt: [follow-up prompt] config: {"model_reasoning_effort": "xhigh", "executor_model": "<actual executor model>", "require_reviewer_model": true}

Prompt fidelity: the manual prompt must be exactly the same text that Codex would receive. Review tracing applies equally to both backends.

Workflow

Step 1: Collect Artifacts (Executor — Claude)

Locate and list these files WITHOUT reading or summarizing their content:

Scan project directory for:
1. Evaluation scripts:    *eval*.py, *metric*.py, *test*.py, *benchmark*.py
2. Result files:          *.json, *.csv in results/, outputs/, logs/
3. Ground truth paths:    look in eval scripts for data loading (dataset paths, GT references)
4. Experiment tracker:    EXPERIMENT_TRACKER.md, EXPERIMENT_LOG.md
5. Paper claims:          NARRATIVE_REPORT.md, paper/sections/*.tex, PAPER_PLAN.md
6. Config files:          *.yaml, *.toml, *.json configs with metric definitions

A verdict-bearing manual response MUST begin with `Reviewer-Model: <exact-model-id>` — pass the model THIS session is actually running as in `executor_model`. Missing, unknown, or same-family identity cannot acquit; emit `REVIEW_UNAVAILABLE` rather than guessing. If the executor model cannot be named, manual review's cross-family claim is unprovable — say so in the report instead of asserting it.

**DO NOT summarize, interpret, or explain any file content.** Only collect paths.

Step 2: Send to Reviewer

Based on the selected reviewer backend (see Reviewer Calling Convention), pass ONLY file paths and the audit checklist to the reviewer. The reviewer reads everything directly.

For `codex`, call `mcp__codex__codex` with:

• `model: gpt-6-astra`
• `config: {"model_reasoning_effort": "ultra"}`
• `sandbox: read-only`
• `cwd: [project directory]`
• `prompt: [the exact full prompt below]`

For `manual`, call `mcp__manual_r