/design-bench

user-invocable: false
description: Run the ABC benchmark — reference URL → design-blind brief → render arms A/B/C → blind judge → scorecard with win-rates over n runs
argument-hint: "[<brief-name>|all] [--n <runs>] [--corpus-version <ver>]"
install: internal

<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios --> <!-- SPDX-License-Identifier: AGPL-3.0-or-later -->

/design-bench — The ABC Benchmark Rig

**Input**: $ARGUMENTS — a seed brief name (`cadence` | `tender` | `slipstream` | `voltage` | `plume`) or `all`, with optional `--n <runs>` (default 3, min 3) and `--corpus-version <ver>`.

This is the **fitness function** for the UI Taste Harness (milestone #13, issue #878). It is built *before* the harness and is the only thing that tells us whether each lever the harness adds actually moves design quality. Running this command answers one question: **did arm A (the harness) score higher against the real reference (arm C) than it did last time?**

Read [`docs/design/abc-benchmark.md`](../docs/design/abc-benchmark.md) (methodology) before running. The three arms, the same-model rule, the C-as-calibration check, the three-layer judging protocol, and the n>=3 rule all live there — this command is the executable workflow over that methodology.

**No application runtime.** ForgeDock is a set of command specs. This command is a step-by-step procedure an agent executes; the only executable helper is the deterministic aggregator `scripts/bench-scorecard.mjs`.

**NEVER use plan mode (EnterPlanMode).**

<!-- FORGE:SPEC_LOADED — design-bench.md loaded and active. Agent is bound by this spec. -->

---

The three arms (one variable)

| Arm | What it is | How this command produces it | |-----|------------|------------------------------| | **A** | ForgeDock harness output | Run the brief through `/design` (#888). **Pluggable** — see [Arm A is pluggable](#arm-a-is-pluggable). | | **B** | Raw frontier model, one shot | Hand the brief to the model once, no tools, no iteration. | | **C** | The real reference page | Capture the brief's real reference URL through the same render pipeline. |

**Hard rule — same generation model for A and B.** The harness is the ONLY difference between A and B; that is what makes the A-vs-B delta attributable to the harness. Different models benchmark models, not the harness.

**Validated generation model: `claude-opus-4-6`.** Benchmark results (#878) show a tier-level quality difference: 90% A-vs-B pairwise win-rate with Opus vs 54% with Sonnet, rubric mean 4.11 vs 3.40, slop 1.2 vs 1.8. The canonical baseline going forward is the Opus run. Raw data: `docs/design/fixtures/runs/full-abc-opus/`. Use Opus for both arm A and arm B unless explicitly testing a different model.

---

Phase 0: Resolve inputs

1. Parse `$ARGUMENTS`: the brief name(s) and `--n` (default 3; **reject n<3** — n=1 is a lie). `all` = every brief under `docs/design/fixtures/briefs/` (excluding `README.md`). 2. For each brief, load `docs/design/fixtures/briefs/<name>.md`. This is the design-blind input handed to arms A and B. 3. Load the brief→reference(C) mapping from `docs/design/fixtures/briefs/README.md`. **The mapping is for the judge/calibration record only — it is NEVER shown to the generating arms.** A real brand name leaking into a brief measures recall, not taste, and inflates arm B.

---

Phase 1: Brief (reference URL → design-blind brief)

The seed briefs are pre-stripped fixtures, so for the seed set this phase is a load, not a derivation.

**To add a new reference URL** (outside the seed set): fetch the page, strip every visual cue (color, type, layout, "make it like X"), keep only what the product *does* / who it's for / what the page must communicate, and assign an **anonymized** product name. Save as a new fixture and add its mapping row. Follow `docs/design/fixtures/briefs/README.md`.

---

Phase 2: Render arms (kill confounds)

Render **all three arms through the identical pipeline** so the judge compares design, not rendering artifacts.

For each of the `--n` runs per brief:

1. **Arm A** — run the brief through `/design` (#888) → self-contained HTML. 2. **Arm B** — one-shot the SAME model with the brief → self-contained HTML. No tools, no iteration. 3. **Arm C** — the real reference URL.

Render every arm with **Playwright** (milestone render dependency) at:

• **1440px** desktop and **390px** mobile,
• `networkidle` + fonts loaded before capture,
• **full-page** screenshot.

For arm C, trim cookie banners / hero video and **record any trim applied** in the run record (reproducibility).

> **Arm A is pluggable.** Until `/design` (#888) lands, arm A is produced by a clearly-labeled manual hook: > generate arm A by hand using the harness method-of-the-day and drop its HTML into the run record marked > `arm_a_source: manual`. The rig — brief loading, render protocol, judging, scorecard — is fully operational > without #888. When #888 lands, replace the manual hook with the `/design` invocation; nothing else changes. > **Do NOT block the benchmark on #888.**

---

Phase 3: Judge (three layers)

The judge is a **named, independent stage**. See [Anti-Goodhart](#anti-goodhart) — it MUST NOT be the harness's own critique loop (#882).

For each run, produce judge output in the [runs-file schema](../docs/design/abc-benchmark.md#scorecard-schema):

Layer 1 — Blind randomized pairwise (primary)

Show the judge two **anonymized** screenshots in **shuffled** order and ask: *"Which reads more like a professionally designed, well-funded SaaS landing page?"* Run **A-vs-C, B-vs-C, A-vs-B**. Record the winner (`"A" | "B" | "C" | "tie"`) per comparison. Screenshots carry no arm labels.

Layer 2 — Rubric (1–5 per arm)

Score each arm on: hierarchy, typography, color discipline, whitespace rhythm, originality-vs-slop, mobile.

Layer 3 — Slop detector (binary)

Run the anti-slop negatives checklist (corpus #880) over each arm; record the count of negative