anti-autoresearch

Anti-Autoresearch 🛡️

· · · · · · · ·

🔬 The field has tolerated unreliable autoresearch long enough — Anti-Autoresearch is the read that finally catches it.

天下苦 autoresearch 久矣 —— Anti-Autoresearch 替研究者们一眼看穿不靠谱的工作。

🏆 Built on a battle-tested foundation: ARIS (~12.5k★ · HuggingFace Daily Papers #1 · 78+ skills across 7+ platforms). Anti-Autoresearch points ARIS's production audit DNA (experiment-audit · paper-claim-audit · citation-audit · kill-argument) outward — auditing a third party's submission instead of your own.

Autoresearch has gone mainstream, and a fast-growing share of what reaches the review pile is machine-generated — and a lot of it doesn't hold up: tables that don't match the text, baselines that aren't there, open-sourced code that won't reproduce its own paper. Reviewers, area chairs, and honest authors increasingly need to verify that, not just suspect it.

Regardless of who or what wrote a paper, does the science hold together and reflect its own evidence? Anti-Autoresearch audits a submission for self-consistency and fabrication, and produces a span-anchored, reviewer-ready report. It is not an opaque AI-text classifier (no authorship probabilities, no "AI-written" verdict) and does not judge misconduct — it surfaces discrepancies a human reviewer should investigate. Separately, it lists transparent, itemized AI writing-style impressions in a quarantined, zero-verdict-weight section (a paper can be integrity-CLEAN while listing many), because reviewers react to them.

🧭 What's inside

46 integrity patterns across 8 families — the coverage vocabulary every finding cites — plus 13 zero-weight AI writing-style impressions and 2 advisories:

Delivered as 11 skills + 1 orchestrating workflow on a deterministic spine: a span-anchored, hashed evidence ledger → LLM auditors that only propose findings → a rules-only reporter that lists every proposal with what the auditor said about it, and summarizes — with 8 patterns eval-gated end-to-end (GRIM · GRIMMER · statcheck · delta arithmetic · hedge-density · …) and the whole gate in CI.

📰 News

• 2026-07-10 — 🧮 Math was invited to overturn false alarms — and honestly declined. We built exact calculators to re-check the arithmetic behind numeric criticals, hoping to auto-clear false alarms like rounding differences. Adversarial review broke every version — the killer was "dropout of exactly 50%" vs a table's 50.4%: a calculator can't know whether a number is exact or rounded. The calculators stayed, demoted: they attach their full working to the report for the human, and nothing gets auto-cleared.
• 2026-07-10 — 🛡️ Criticals must now survive their own defense. A critical has to state which innocent explanations it ruled out (blank → drops to major), face one fresh refutation thread whose only power is a ⚠️ CONTESTED marker for the human, and — if numeric — declare exactly which numbers it computes over. Reports stamp adjudicator: deterministic-rules-v2. ⚠️ Re-run the auditors on old findings files instead of re-adjudicating them.
• 2026-06-28 — ✍️ v0.5 — the AIS track. 13 itemized AI writing-style impressions now live in a separate, zero-verdict-weight report section — a paper can be integrity-CLEAN while listing many. Transparent impressions, never an authorship verdict; the 5 pure-style patterns moved out of family F.

• 2026-06-27 — 🧬 v0.4 — the taxonomy grows to 8 families (new: proof & derivation G, evaluation design H — the full map is in What's inside above). The deterministic eval gate grew 3→8 patterns (GRIM / GRIMMER / statcheck + a conservative hedge-density screen), CI landed, and prior-art acknowledgments were added.
• 2026-06-26 — 🚀 v0.1 — initial release. Reviewer-side integrity forensics for autoresearch / AI-Scientist papers: evidence ledger, deterministic adjudicator, observability tiers. Not an AI-text detector.

🚀 Quickstart

Agent workflow (normal use)

Anti-Autoresearch runs as a Claude Code skill workflow — the Python tools are the deterministic spine inside that workflow, not the usual interface.

# 1) Install the skills + workflow (global, or pass a project's .claude/skills dir)
git clone https://github.com/wanshuiyin/Anti-Autoresearch.git
./Anti-Autoresearch/tools/install_anti_autoresearch.sh              # → ~/.claude/skills
# project-local instead: ./Anti-Autoresearch/tools/install_anti_autoresearch.sh ./.claude/skills

# 2) Wire the cross-model reviewer (end state: Claude Code exposes mcp__codex__codex)
claude mcp add codex -- codex mcp-server
claude mcp list

# 3) Audit a paper
claude
> /anti-autoresearch ~/papers/submission

The run writes REPORT.md + report.json + claims.json + per-skill *.findings.json into the paper directory. Put the code/result artifacts alongside the paper to unlock L2 checks; PDF/source-only runs are observability-limited by design.

Zero verdict weight — the AIS + advisory tracks (reported, never moves the verdict)

Three skills produce outputs that are reported but carry zero weight on the integrity verdict — the non-integrity categories that round out a report: the AIS writing-style track and the advisory memos. They matter to a human reviewer (a style impression, the worst-case rejection paragraph, prior-art overlap), so the report shows them in their own section — but the deterministic verdict stays driven only by the 46 integrity patterns. A paper can be CLEAN_GIVEN_EVIDENCE while listing many. /anti-autoresearch runs them automatically; to run one standalone, build the ledger first (next section) and invoke it like any auditor.

Single-skill use

Every auditor is also a standalone skill — the installer drops all of them plus the workflow, so you can run just the axis you care about. They share one contract, so run it in order:

claude
# 1) Build the evidence ledger ONCE — the spine every auditor anchors to. Skip it and
#    any auditor stops with:  NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST
> /evidence-ledger ~/papers/submission        # → claims.json + observability level (L0/L1/L2)

# 2) Then run any auditor below against that ledger → <skill>.findings.json

The verdict-bearing auditors — each takes the paper dir, reads the ledger, and proposes span-anchored findings the deterministic adjudicator turns into the verdict (the zero-weight AIS + advisory skills are in the section above):

A single skill only proposes span-anchored findings — it never returns a verdict. To get one, feed the findings to the deterministic adjudicator (the python3 tools/adjudicate_findings.py … --ledger … command in the next section); the model never grades. Two more notes: consistency-audit, presentation-signals, and ai-style-impressions also write a *.deterministic.findings.json (works with no cross-model reviewer wired); and /anti-autoresearch runs every auditor above in one shot, adding ingest (arxiv-id / pdf → workdir + pdftotext), automatic observability, auto-selection of which auditors apply, and the final cross-dimension verdict + REPORT.md.

Deterministic core (CI / offline / zero-dependency)

This bypasses the agent layer and exercises only the eval-tested deterministic checks — use it for CI, regression tests, or environments with no cross-model reviewer (Python 3 stdlib, nothing to install):