/eval-design-forensics

name: eval-design-forensics
description: "Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/result files. A fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per pass) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Leakage and under-reporting are usually HONEST methodological errors — every finding describes a discrepancy to CHECK, never an accusation. An LLM generating GROUND-TRUTH labels is HP-FAKE-GT (experiment-forensics) — routed there, not here. Emits eval-design-forensics.findings.json; computes NO verdict. Detect-only. Triggers: \"eval design audit\", \"evaluation validity\", \"train/test leakage\", \"data leakage\", \"is the score measuring generalization\", \"LLM judge bias\", \"is the judge validated\", \"selective reporting\", \"cherry-picked results\", \"评估设计审计\", \"评测有效性\", \"数据泄漏\", \"训练测试集泄漏\", \"裁判模型有没有验证\", \"选择性报告\"."
argument-hint: [paper-dir | claims.json]
allowed-tools: Bash(*), Read, Write, Grep, Glob, WebSearch, WebFetch, mcp__codex__codex

Eval-Design Forensics — does the evaluation measure what the paper claims?

Audit evaluation-design and reporting validity for: **$ARGUMENTS** (requires `claims.json` from `/evidence-ledger`). Emit span-anchored `eval-design-forensics.findings.json`. This skill computes **no verdict**.

> 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is > verdict-bearing input — it proposes the findings the deterministic adjudicator > turns into the report. Re-firing it on a wall-clock timer adds no signal: its > output changes only when the **paper / ledger** changes (or a repo arrives, > raising the observability level), not with the clock. Schedule the *external wait > that precedes it* — ledger built (or artifacts released → L2) → audit **once**. > (Mirrors ARIS's external-cadence doctrine.)

> Adapted from the ML-evaluation-methodology literature — the leakage taxonomy of > Kapoor & Narayanan (2023), the LLM-as-judge validity work (MT-Bench > self-enhancement, self-preference, position bias), and the "Show Your Work" / > reproducibility-checklist reporting norms — reframed to audit a **third party's** > evaluation. A favourite autoresearch shortcut is to report a number that is > arithmetically self-consistent (family A), runs real code against a real ground > truth (family D), and **still does not measure what it claims**: the protocol > leaks, the load-bearing metric is a conflicted/unvalidated LLM judge, or the > reporting quietly drops a declared condition. This skill is the constraint that > asks "is this a *valid measurement of the claim*?", pointed at a submission, and > it stays honest — leakage and under-reporting are usually **honest methodological > errors**, so every finding is a discrepancy to *clarify*, never an accusation.

Why this exists

An optimizing pipeline (or rushed human) treats the evaluation as a number to make go up, not a measurement to keep valid. The repeatable failure modes — distinct from "is the number real?" (family D) — are:

• **Leakage** — the train/test boundary is broken (preprocessing fit before the

split, no held-out set, duplicates across splits, a random split over time-ordered data, the same subject in both splits, an evaluated LLM that saw the benchmark in pretraining), so the reported score may not measure **generalization** at all. `HP-EVAL-LEAKAGE`

• **Judge validity** — the headline rests on an automatic **LLM judge** that is

*conflicted* (the same model/family as a compared system, so its preference for that system is the "evidence") or *unvalidated* (no human-agreement correlation, no position/length bias control). `HP-JUDGE-VALIDITY`

• **Selective reporting** — a dataset / baseline / metric / seed-count the setup

**explicitly declares** is dropped from the results, the metric is **switched** across tables to keep the method ahead, or "we report the best run/prompt/ checkpoint" with **no held-out selection set** (selecting on the test set). `HP-SELECTIVE-REPORTING`

None of these is inherently misconduct — they are what an agent does when nothing forces a *valid* evaluation. The **stated** version is decidable at **L0/L1** from the described protocol; the **verified** version (real split/preprocessing/result files) deepens at **L2**. What this skill will **not** do is *guess*: three leakage subtypes are undecidable even with the repo and are handed off as `needs_external_check`, not invented (see below).

Core principle

**Ledger-anchored, span-verified, reviewer≠adjudicator, honest about what it cannot settle.** Four properties:

1. **Anchor to a PAPER claim.** Every above-`info` finding cites a ledger `claim_id` and quotes a **verbatim span of that claim's `text_span`** (`references/integrity-forensics-contract.md` rules 1–2). The leak/judge/reporting tell lives in the *protocol / setup-description* — usually `method` and `scope` claims, with `comparison` / `number` for the ju