/weak-agent-test

name: weak-agent-test
description: "Run the weak-agent adversarial test harness against docx-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe, or a local agent harness's pre-produced runs) to perform real document tasks over six scenarios — five editing (MNDA form-fill + font fidelity, invoice table-edit/restructure + logo replace, résumé styling, contract redlining + commenting, contract finalize via accept/reject + comment reply/resolve) and one authoring (T. S. Eliot poetry journal: multi-column, verse, footnotes, links, figure) — renders every result with Word, has opus judge them against ground-truth rubrics, measures each exercise's tool economy, token cost, wall-clock, and correctness (from transcripts for Claude, the exercise.json ledger for the local harness), and synthesizes a prioritized ergonomics report. Use when the user says 'adversarial review', 'test docx-cli with weak agents', 'run the haiku harness', 'weak agent test', or wants to re-run yesterday's adversarial process."
allowed-tools: Bash, Read, Write, Glob, Workflow
metadata:
  internal: true

Adversarial review — weak-agent harness for docx-cli

This harness answers one question: **can weak agents actually use docx-cli to get real work done, and what should we fix first?** It runs the `weak-agent-test` workflow (`.claude/workflows/weak-agent-test.js`), which fans out one weak exercise agent per scenario (Haiku by default — swappable to Sonnet via `args.model`), renders every output with Microsoft Word, grades each against ground-truth criteria with an **opus** judge, and has **opus** synthesize a prioritized improvement report. Exercise agents do NOT self-report tool counts — every tool-economy and token number is **measured** after the run (agents under-count their own calls ~2×, so self-reports were dropped): from the agent transcripts for the Claude arms, from each scenario's `exercise.json` ledger for the local arm. Both roll up into the same Run-metrics table (tokens, wall-clock, tool split, correctness) via `exercise-metrics.ts`.

The test corpus is **bundled with this skill** under `scenarios/`, one folder per scenario, named after its key (`scenarios/mnda/`, `scenarios/invoice/`, …). Each scenario folder is self-describing and holds everything that scenario needs:

• `task.md` — the AGENT-FACING request, written as a human delegating the work:

the goal, the data, the intent — and **no tool vocabulary** (no `docx` commands, locators, or OOXML terms), because discovering which features deliver the outcome is part of what's measured,

• `criteria.md` — the JUDGE-ONLY grading rubric (the precise, tool-specific checks).

The stage step **withholds it from the agent's run workspace**, and the judge reads it from the pristine source — the agent never sees the answer key,

• the fixture `.docx` to work on (edit scenarios only; authoring scenarios create

their output fresh),

• `assets/` — any additional inputs (data files, images; empty for most edit

scenarios).

The workflow's `SCENARIOS` manifest holds only the per-scenario **routing** metadata (key, bucket label, `edit`/`author` kind, the doc filename); whether a baseline gets rendered is DERIVED from the kind (every `edit` scenario has a pristine source, so it gets one — see `hasBaseline()`), not a stored field. The actual request/criteria/fixture/assets all live in the folder. The skill is therefore self-contained and travels with its test corpus. To change what a scenario tests, edit the files in its folder. (Heavy, ephemeral run outputs — edited docx, renders, reviews, the report — are dumped to `./tmp/docx-weak-agent-test/<ts>/`, never into the repo.)

Staging is ONE code path for every backend: `scripts/stage-scenario.ts` copies a scenario folder, strips the judge-only `criteria.md`, and verifies the inputs landed. The workflow's Stage agent runs it per scenario; the local corpus runner imports it.

Each run produces, under the timestamped run dir, **one result folder per scenario** (named after its key) plus the run-level report and metrics:

<RUN_DIR>/
  REPORT.md            ← synthesized report + the measured run-metrics section, both
                          written by your post-run exercise-metrics.ts --append-report
  exercise-metrics.md  ← measured per-exercise-agent tokens/time/tool split
  exercise-metrics.json
  <key>/               ← one per scenario; the worked-on copy lives here
    task.md  assets/   ← (criteria.md is withheld from this copy — judge-only)
    scratch/           ← the agent's PRIVATE scratch dir (batch JSONL, drafts). The
                          session scratchpad is shared by every concurrent agent, so
                          same-named files there collide — see "Isolation breaches"
    <doc>.docx         ← the edited/authored document
    renders/output/    ← the OUTPUT: Word-rendered page PNGs + read.md (markdown read view)
    renders/baseline/  ← the pristine "before": page PNGs + read.md (every EDIT scenario;
                          absent only for the authored eliot-journal — no source to diff)
    review.md          ← the judge's saved review for this task (written in-run)
    verdict.json       ← the judge's structured verdict incl. taskSuccess (written in-run
                          by the judge — the correctness source the metrics pass reads)
    metrics.json       ← this task's measured tokens/time/tool split + correctness
                          (written by your post-run metrics pass)

The **render step fires the moment each task finishes** (for both arms) and produces, for the OUTPUT and — whenever a pristine source exists (every edit scenario) — its BASELINE "before", BOTH deliverables in each render dir: the page PNGs AND a `read.md` (the markdown read view of that doc). The judge reads all four (output PNGs + read.md, baseline PNGs + read.md) to compare before/after both visually and textually. The workflow's render step is **idempotent**: for the **loc