/audit-engine

name: audit-engine
description: >
  Activate when the user wants to audit a paper's empirical or technical claims
  against a linked code repository — checking whether experiments, datasets,
  models, metrics, and hyperparameters described in the paper actually exist
  and match the code. Produces a structured audit report classifying each claim
  as CONFIRMED, PARTIAL, MISSING, or MISMATCH, with file/line evidence. Useful
  for reproducibility checks, reviewer due diligence, and pre-submission
  self-audits of ML/CS/empirical papers that ship code.

> **Orchestration Log**: When this skill is activated, append a log entry to `outputs/orchestration_log.md`: > ``` > ### Skill Activation: Audit Engine > **Timestamp:** [current date/time] > **Actor:** AI Agent (audit-engine) > **Input:** [paper + repo being audited] > **Output:** [brief summary — e.g., "Audited 18 claims: 12 CONFIRMED, 3 PARTIAL, 2 MISSING, 1 MISMATCH"] > ```

Audit Engine

Core Principle

Papers make claims. Code embodies what was actually done. This engine systematically checks whether the two agree. For every empirical or technical claim in the paper — datasets used, models trained, metrics reported, hyperparameters set, ablations run — the engine locates supporting evidence in the linked repository and classifies the match.

This is the complement to `verification-engine`, which checks citations against external sources. Audit-engine checks *the paper's own claims* against *the paper's own code*. Together they cover both failure modes of LLM-assisted writing: mis-cited prior work and unsupported own-work claims.

Inspired by the `/audit` command in the Feynman research agent (Companion AI, 2026), adapted to the IS/CS methodological style of this plugin.

When to Activate

• User says "audit the paper", "check paper vs. code", "verify my experiments",

"reproducibility audit", "does the code match what I wrote"

• Before submitting a paper with an accompanying code release
• Before open-sourcing the repo of a published paper
• When reviewing someone else's paper + artifact
• As optional Phase 7.5 of the paper machine pipeline (after verify-citations,

before prepare-submission)

When NOT to Activate

• The paper has no code artifact (pure theory, position paper, qualitative study

without computational analysis) → say so and exit

• The user wants to verify *citations* → activate `verification-engine` instead
• The user wants to check *writing quality* → activate the writing-engine

`/analyze-writing` command

---

Inputs

Required: 1. **Paper source** — `paper.tex`, `draft.md`, or explicit `$ARGUMENTS` path 2. **Code repository** — one of:

• Local path (`./experiments/`, `~/repos/myproject`)
• GitHub URL (clone or use `gh repo view` / `WebFetch` on raw files)
• Archive link (Zenodo, OSF) — ask user to download locally first

If the repo location is not supplied, scan the paper for common signals:

• "Code available at [URL]" / "Our implementation is at [URL]"
• GitHub URLs in footnotes or acknowledgements
• A `code_availability` section
• A `REPRODUCIBILITY.md`, `ARTIFACT.md`, or similar file sibling to the paper

If still not found: ask the user once, then exit.

---

Step 1: Extract Auditable Claims

Scan the paper for claims that can be checked against code. Ignore claims that are purely conceptual, historical, or theoretical.

Claim Categories (check in order)

| Category | What to look for | Priority | |----------|------------------|----------| | **Dataset** | Named datasets, split sizes, sample counts, data sources | HIGH | | **Model** | Model names, architectures, parameter counts, checkpoints | HIGH | | **Training** | Epochs, batch size, learning rate, optimizer, hardware | HIGH | | **Metrics** | Reported numbers (accuracy, F1, BLEU, loss values, percentages) | HIGH | | **Experiments** | Named experimental conditions, ablations, baselines | HIGH | | **Hyperparameters** | Specific values in tables or "Training Details" | MEDIUM | | **Preprocessing** | Tokenization, normalization, filtering steps | MEDIUM | | **Evaluation** | Test protocol, prompt templates, judge models, seeds | MEDIUM | | **Infrastructure** | GPUs, training time, framework versions | LOW | | **Figures** | Plots claimed to come from "our experiments" | MEDIUM |

Extraction Pattern

For each claim, record:

{
  id: "C01",
  category: "Model",
  section: "4.2 Model Training",
  claim_text: "We fine-tune LLaMA-3-8B for 3 epochs with a learning rate of 2e-5.",
  testable_facts: [
    "model == LLaMA-3-8B",
    "epochs == 3",
    "learning_rate == 2e-5"
  ],
  priority: "HIGH"
}

Claims with concrete numbers, names, or identifiers are testable. Vague claims ("we use a standard transformer") are not auditable — mark them as `NOT_AUDITABLE` and skip.

**Output:** `outputs/audit_claims.md` — numbered list of all testable claims.

---

Step 2: Map the Repository

Before searching, build a lightweight mental map of the repo. Do not read every file.

1. **Top-level listing** — `Glob` on `**/*.{py,ipynb,yaml,yml,json,toml,sh,md}` at depth 2-3 2. **Identify key files** by name convention:

• `train.py`, `main.py`, `run_experiments.py`, `eval.py` → entry points
• `config.yaml`, `hparams.json`, `sweep.yaml`, `*.toml` → configuration
• `requirements.txt`, `pyproject.toml`, `environment.yml` → dependencies
• `README.md`, `REPRODUCE.md`, `docs/` → documentation
• `results/`, `outputs/`, `logs/`, `wandb/` → experiment artifacts
• `datasets/`, `data/`, `load_data.py` → data loaders

3. **Detect framework** — PyTorch, JAX, TensorFlow, HuggingFace, scikit-learn — this guides search patterns 4. **Detect experiment tracking** — wandb, mlflow, tensorboard, plain CSV logs

Record this as an internal map; do not output it unless the user asks.

---

Step 3: Search for Evidence (per claim)

For each testable claim, systematically search for supporting code evidence.

Search Strategy

Use `Grep