/baseline-comparison-audit

name: baseline-comparison-audit
description: "Audit whether a paper's baseline comparisons are COMPLETE, FAIR, and SIGNIFICANT: a required recent SOTA baseline is missing while 'best/SOTA' is claimed (HP-MISSING-BASELINE); a baseline is undertuned / given less compute-tuning-data, run at a mismatched config, or the equal-budget ablation-as-baseline is absent (HP-WEAK-BASELINE); 'outperforms' is asserted over overlapping error bars or with no variance/seeds (HP-SIG-OVERLAP); and a cross-row 'improves over baseline by X%' is arithmetically wrong (HP-DELTA-ERROR, cross-row form only). A versioned per-domain baseline profile + a live leaderboard/recency search are assembled by the EXECUTOR as structured facts; a fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per dimension) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Works at L0 (stated comparisons) and deepens at L2 (configs/result files). A completeness question it cannot settle internally becomes needs_external_check, never a guessed missing baseline. Emits baseline-comparison-audit.findings.json; computes NO verdict. Detect-only. Triggers: \"baseline audit\", \"missing baselines\", \"is the comparison fair\", \"weak baseline\", \"baseline 误报\", \"SOTA earned?\"."
argument-hint: [paper-dir | claims.json]
allowed-tools: Bash(*), Read, Write, Grep, Glob, WebSearch, WebFetch, mcp__codex__codex

Baseline Comparison Audit — is the comparison complete, fair, and significant?

Audit baseline-comparison integrity for: **$ARGUMENTS** (requires `claims.json` from `/evidence-ledger`). Emit span-anchored `baseline-comparison-audit.findings.json`. This skill computes **no verdict**.

> 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is > verdict-bearing input — it proposes the findings the deterministic adjudicator > turns into the report. Re-firing it on a wall-clock timer adds no signal: its > output changes only when the **paper / ledger** (or the live leaderboard it > cross-checks) changes, not with the clock. Schedule the *external wait that > precedes it* — ledger built → audit **once**. (Mirrors ARIS's external-cadence > doctrine.)

> Adapted from ARIS `paper-claim-audit` — its **scope-overclaim** and > **delta-arithmetic** checks, reframed from "paper vs result files" to **"is the > SOTA claim earned, and is the comparison a fair fight?"** — plus a per-domain > baseline profile and a completeness / fairness / significance split. A favourite > autoresearch shortcut is to claim SOTA while omitting the obvious recent baseline, > to beat an undertuned one, or to write "outperforms" over error bars that overlap. > This skill is the constraint that asks for the fair fight, pointed at a third > party's submission, and it stays honest about what it cannot settle from a PDF.

Why this exists

An autoresearch pipeline (or rushed human) optimises for the *headline* and treats the comparison table as scaffolding to fill, not a fair experiment to run. The repeatable failure modes:

• **Completeness** — "achieves state-of-the-art on GSM8K" while the obvious recent

baseline a 2024–2026 reviewer expects is simply absent from the table, or the strong classical **floor** (BM25 for retrieval, GBDT for tabular, a linear/naive forecaster for time-series) is skipped while only weak neural baselines are beaten. `HP-MISSING-BASELINE`

• **Fairness** — the proposed method is tuned for 100 epochs / 5 seeds / extra data,

the baseline is run at default settings for 10; or the compared rows use different backbones, splits, or eval protocols; or the single most informative baseline — the method's **own backbone with the new component removed, at an identical budget** — is missing. `HP-WEAK-BASELINE`

• **Significance** — "consistently outperforms" on a 0.3-point gap with overlapping

error bars, with no variance / no seed count reported at all, or resting on a single dataset too thin for the "consistent / across-the-board" wording. `HP-SIG-OVERLAP`

• **Delta arithmetic** — "improves over the strongest baseline by 16%" when the

baseline row is 73.1 and the proposed row is 78.0 (+6.7% relative / +4.9 points), the two operands sitting in *different* cells so the single-sentence deterministic pass cannot pair them. `HP-DELTA-ERROR` (cross-row form)

None of these is inherently misconduct — they are what an optimizing agent does when nothing forces a fair comparison. The *stated* version is decidable at **L0** from the manuscript; the *verified* version (real configs, real seeds) deepens at **L2**. What this skill will **not** do is *guess*: where no domain profile exists and the leaderboard search is inconclusive, the completeness question is handed off as `needs_external_check`, not invented.

Core principle

**Ledger-anchored, span-verified, reviewer≠adjudicator, honest about what it cannot settle.** Four properties:

1. **Anchor to a PAPER claim.** Every above-`info` finding cites a ledger `claim_id` and quotes a **verbatim span of that claim's `text_span`** (`references/integrity-forensics-contract.md` rules 1–2). The "outperforms / SOTA / best / first" language lives in `comparison` and `scope` claims; the reported baseline set lives in `baseline` claims; values/table rows in `number` / `table_cell` claims. The anchor is whichever paper claim the finding undermines — the expected-baseline list, a leaderboard URL, or a config `file:line` are **forensic context for the description**, never the anchor. 2. **The executor assembles facts; the reviewer judges.** The profile + a live `WebSearch`/`WebFetch` give a *candidate expected-baseline set with sources and dates*; the executor passes it as **structured input** and never pre-declares "baseline X is missing" (`references/reviewer-independence.md`). The model **proposes**; `tools/adjudicate_findings.py` **decides**. This skill computes **no