/consistency-audit

name: consistency-audit
description: "Flagship intra-paper self-consistency forensics: does the paper contradict ITSELF across abstract/intro/tables/body/appendix, and does the method DESCRIBED match the method EVALUATED? Needs no external ground truth — works PDF-only (L0). Runs a deterministic arithmetic pass + a fresh cross-model semantic pass, every finding span-anchored to the evidence ledger (claims.json), reviewer≠adjudicator. Emits consistency-audit.findings.json; NEVER computes the verdict. Triggers: \"consistency audit\", \"check the paper against itself\", \"self-consistency\", \"内部自洽\"."
argument-hint: [paper-dir | claims.json]
allowed-tools: Bash(*), Read, Write, mcp__codex__codex

Consistency Audit — the paper vs itself

Audit intra-paper self-consistency for: **$ARGUMENTS** (requires `claims.json` from `/evidence-ledger`). Emit span-anchored `consistency-audit.findings.json`; the deterministic adjudicator — not this skill — computes the verdict.

> 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is > verdict-bearing input — it proposes the findings the deterministic adjudicator > turns into the report. Re-firing it on a wall-clock timer adds no signal: its > output changes only when the **paper / ledger** changes, not with the clock. > Schedule the *external wait that precedes it* — ledger built → audit **once**. > (Mirrors ARIS's external-cadence doctrine.)

> The flagship instrument. Internal contradiction is the single most defensible > thing you can check on an unknown submission: it needs no external GT, runs at L0 > (PDF-only), and is exactly where machine-generated papers crack — they hallucinate > *local* coherence. Recall scales with the ledger: a PDF-text (L0) ledger extracts > only number/scope spans, so the table/caption/method-drift checks gain teeth at L1 > (LaTeX), where tables and captions are actually extracted. Adapted from ARIS > `paper-claim-audit`, reframed from "paper vs result files" to **"paper vs > itself."** There is no external ground truth in this skill.

Why this exists

An autoresearch pipeline (or rushed human) writes the abstract, the tables, the method section, and the appendix in separate passes and never reconciles them. The result is a paper that disagrees with **itself**:

• abstract quotes 85.3% accuracy; the best row of its own Table 2 is 84.7%;
• "improves by 16%" when 73.1 → 78.0 is +6.7% relative / +4.9 points;
• "mean over 5 seeds" where the number is the single best seed, and N=3 in the table;
• method section says "no test-time labels"; the experimental-setup paragraph loads

gold labels for calibration;

• "comprehensive evaluation across diverse benchmarks" on two datasets, one domain.

None of this needs the code, the data, or a single external fact to detect — only the paper, read against itself. That is why this skill is the flagship and the only auditor that always runs: it **runs at L0** (no external GT), the level at which we have the least to work with — though L0/PDF-text recall is bounded by what the extractor recovers (number/scope spans only); the richer table/caption checks need the L1 LaTeX ledger.

Core principle

**Ledger-anchored, span-verified, paper-vs-itself, reviewer≠adjudicator.** Two passes feed the pipeline:

1. a **deterministic** arithmetic pass (no model) — reproducible, the eval backbone; 2. a **fresh cross-model** semantic pass — meaning drift the arithmetic can't see.

Both emit findings conforming to `schemas/finding.schema.json`. **Every above-info finding cites a ledger `claim_id` + a verbatim span** (`references/integrity-forensics-contract.md` rules 1–2). The model **proposes**; `tools/adjudicate_findings.py` **decides** (`references/reviewer-independence.md` Layer 2). This skill computes **no verdict**.

How this differs from the other auditors (route correctly)

| Auditor | Question it answers | Level | |---------|---------------------|------| | **`consistency-audit`** (this) | **Does the paper contradict ITSELF / does described method = evaluated method?** | **L0 — intra-paper, no external lookup** | | `experiment-forensics` | Are the reported numbers what the code actually computes? (fake GT, self-norm, phantom) | L2 | | `baseline-comparison-audit` | Are the right baselines present, tuned, and is "SOTA" earned? | L0 stated / L2 verified | | `citation-forensics` | Do the cited papers exist and support the claim made? | L0 | | `proof-derivation-forensics` | Does the WRITTEN proof/derivation actually establish its theorem? (gap / circularity / invalid step / symbol drift / smuggled assumption) | L1 | | `presentation-signals` | Surface "AI-flavor" hints (auxiliary, surface-class) | L0 | | `adversarial-case-builder` | Strongest evidence-bound rejection memo (no verdict weight) | any |

**Do NOT raise here** (hand off instead): code/result-level fraud → `experiment-forensics` (needs L2); "first / SOTA / beats prior work" external truth → `baseline-comparison-audit`

• `citation-forensics` (emit `needs_external_check`); citation existence/context →

`citation-forensics`; proof / derivation validity (a theoretical relation that must be *derived*, not measured — gap / circularity / invalid step / symbol drift / smuggled assumption) → `proof-derivation-forensics` (family G); surface/AI-flavor → `presentation-signals`; the rejection memo → `adversarial-case-builder`. This skill never reaches outside the paper.

Constants & Reviewer Calling Convention

REVIEWER_MODEL          = gpt-5.6-sol                  # different family from executor (Claude)
REVIEWER_REASONING      = xhigh                    # always; effort never lowers reviewer quality
REVIEWER_SANDBOX        = read-only                # detect-only; never mutate the paper
REVIEWER_CWD            = <paper-dir>              # so it can read claims.json + sources directly
THREAD_POLICY           = fresh mcp__codex__codex per run; NEVER mcp__codex__codex-reply
TAXONOMY_VERSION        = 0.5