/paper-claim-audit

name: paper-claim-audit
description: "Zero-context verification that every number, comparison, and scope claim in the paper matches raw result files. Uses a fresh cross-model reviewer with NO prior context to prevent confirmation bias. Use when user says \"审查论文数据\", \"check paper claims\", \"verify numbers\", \"论文数字核对\", or before submission to ensure paper-to-evidence fidelity."
argument-hint: "[paper-directory]"
allowed-tools: Bash(*), Read, Write, Edit, Grep, Glob, Skill, mcp__codex__codex, mcp__antigravity__antigravity, mcp__antigravity__antigravity-reply, mcp__grok__grok, mcp__grok__grok-reply, mcp__claude-review__review, mcp__claude-review__review_reply, mcp__claude-review__review_start, mcp__claude-review__review_reply_start, mcp__claude-review__review_status

Paper Claim Audit: Zero-Context Evidence Verification

> **Before each reviewer slot**, follow [Selectable Reviewers](../shared-references/reviewer-backends.md): > resolve invocation → `.aris/reviewer.json` → Claude Code default (Codex MCP) > with `reviewer_route.py` through the canonical helper resolver, preserving this > slot's tier, fresh contexts, source/image inputs and deterministic gates. > Codex examples/pins below apply only to `backend=codex`; other routes use their > own first-call/reply contract. Save the route and actual returned identity; > unavailable selections are `REVIEW_UNAVAILABLE`. Classify acceptance from actual > families/completion/scope. Forward invocation reviewer/model/effort overrides > to reviewer-bearing children. Generation workers are unchanged.

> 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is > verdict-bearing — it judges paper-to-evidence fidelity with a deliberately > zero-context fresh reviewer. Re-firing that verdict on a wall-clock timer adds > no new signal (it changes only when the *paper or results* change). Schedule > the *external wait that precedes it* — paper draft ready → then audit > **once**. See > [`shared-references/external-cadence.md`](../shared-references/external-cadence.md).

Verify that every claim in the paper matches raw evidence for: **$ARGUMENTS**

Why This Exists

The executor writes experiments AND writes the paper. It "knows" what the results should be. This creates confirmation bias:

• Rounding 84.7% up to 85.3%
• Reporting best seed instead of average
• Citing metrics from a different experiment config
• Claiming "improves by 15%" when the delta is actually 12.8%

A **fresh reviewer with zero prior context** catches these because it has no expectations — it just compares paper text vs raw files.

How This Differs From Other Audit Skills

| Skill | Question it answers | |-------|-------------------| | `/experiment-audit` | Is the experiment code honest? (fake GT, normalization fraud) | | `/result-to-claim` | Does the data scientifically support this claim? | | **`/paper-claim-audit`** | **Does the paper report the data truthfully and precisely?** |

Core Principle

**Zero-context, fresh reviewer.** The auditor receives ONLY:

• Paper .tex files (the claims)
• Raw result files (the evidence)

It does NOT receive:

• ❌ EXPERIMENT_LOG.md
• ❌ EXPERIMENT_TRACKER.md
• ❌ AUTO_REVIEW.md
• ❌ NARRATIVE_REPORT.md
• ❌ Any executor summary or interpretation
• ❌ Any prior audit results
• ❌ Any conversation history

This is **stricter than reviewer-independence** — it's zero-context evidence audit.

Workflow

Step 1: Collect Files (Executor — Claude)

Locate paper and result files WITHOUT reading or interpreting them.

**Paper files** (claims) — paths shown relative to the shell's working directory so you can find them with `ls`; when writing them into `audited_input_hashes`, use paths relative to the paper dir (no `paper/` prefix) per the "Submission Artifact Emission" section below:

paper/main.tex                # → hash key: main.tex
paper/sections/*.tex          # → hash key: sections/*.tex
paper/tables/*.tex (if separate)   # → hash key: tables/*.tex

**Result files** (evidence):

results/*.json, results/*.jsonl, results/*.csv, results/*.tsv
outputs/*.json, outputs/*.csv
wandb-summary.json (if exists)
**/metrics.json, **/eval_results.json
**/config.yaml, **/args.json (experiment configs)

**Exclude** (no summaries, no interpretations):

EXPERIMENT_LOG.md, EXPERIMENT_TRACKER.md, AUTO_REVIEW*.md
NARRATIVE_REPORT.md, PAPER_PLAN.md, findings.md
Any .md file that is an executor-written summary

Step 2: Fresh Reviewer Audit (GPT-6-Astra — NEW thread, no reply)

**CRITICAL: Use `mcp__codex__codex` (new thread), NEVER `mcp__codex__codex-reply`.** Every run must be a fresh context.

> Resolve this reviewer slot first; the following is the `backend=codex` example.

mcp__codex__codex:
  model: gpt-6-astra
  config: {"model_reasoning_effort": "ultra"}
  prompt: |
    You are a paper-to-evidence auditor. You have ZERO prior context about
    this research. You will receive only paper source files and raw result
    files. Your job is to verify that every number in the paper exactly
    matches the raw evidence.

    Paper files to read:
    [list .tex file paths]

    Result files to read:
    [list .json/.csv/.yaml file paths]

    ## Audit Protocol

    ### A. Extract Every Quantitative Claim
    For each number, percentage, comparison, or scope statement in the paper:
    - Location (section, table, caption, or inline text)
    - Exact claim text
    - The number or comparison being made

    ### B. Trace Each Claim to Evidence
    For each extracted claim, find the supporting raw data:
    - Which result file contains this number?
    - What is the EXACT value in that file?
    - Match status: exact_match / rounding_ok / mismatch

    ### C. Check These Specific Failure Modes

    1. **Number inflation**: Paper says 85.3%, raw file says 84.7%
       Rule: only standard rounding to displayed precision is allowed

    2. **Best-seed cherry-pick**: Paper says "achieves 90.2%" but
       that's the