/ab-equivalence

name: ab-equivalence
description: "Compare two versions of an LLM-directed document - an original (teacher) and a candidate (student) - across a transfer set and return a per-case behavioural-equivalence verdict plus an efficiency signal. A transform-agnostic library capability other skills compose to gate a transform on behavioural sameness. TRIGGER when asked to A/B test two versions of a prompt / instruction / skill, to check whether a rewritten or compressed document still behaves the same as the original, to validate behavioural equivalence between two document versions, to gate a transform on no-regression, or when a skill needs the run-the-runner-on-both-versions-and-judge-equivalence capability."

A/B equivalence - a transform-agnostic behavioural-equivalence capability

A thin capability that compares two versions of an LLM-directed document - an `original` (the teacher) and a `candidate` (the student) - across a transfer set, and returns a per-case verdict on whether the candidate still induces the behaviour the original induced.

It is **transform-agnostic**: it judges *behavioural equivalence between two versions* and neither knows nor cares which transform produced the candidate. It therefore serves every optimizer transform that claims to preserve behaviour - compression today, directive-clarity next - not just compression. It is a **library capability** other skills compose: `semantic-compress` invokes it to gate a distillation, and `skill-forge` exposes it alongside its own quality gate. It does **not** judge *absolute quality* ("is this skill good?") - that is a different question answered by different judges. A/B equivalence judges *sameness between two versions* ("does the candidate still do what the original did?").

This skill **owns the runner** (`references/runner-prompt.md`, the pure-wrapper template, paths relative to this skill directory). The runner is the shared execution primitive: it applies one version of a document to one case input and returns a transcript and self-report. Skills that need behavioural comparison compose this capability rather than re-implementing the runner.

The runner ships in **two variants**, both pure wrappers returning the same six self-report fields (see `references/runner-prompt.md`):

• the **skill variant** (default) - the document is invoked on demand against a case input;
• the **instruction-file variant** - the document is an *always-loaded* agent instruction file (`CLAUDE.md`, `AGENTS.md`, `GEMINI.md`, `.cursor/rules/*`, `.github/copilot-instructions.md`); the runner is handed only that file as its operating context plus a realistic repo task, and runs **read-only / sandboxed** (it states the actions it would take, never mutating the repo).

The variant is the caller's choice (skill-forge's artifact-type detection selects it); both produce a transcript the equivalence judge and skill-forge's lenses read identically.

Input contract

| Input | Required | Notes | |-------|----------|-------| | `original` | yes | Path to the teacher document - the version whose behaviour is the equivalence target. | | `candidate` | yes | Path to the student document - the transformed version under test. | | `transfer_set` | yes | Array of cases spanning the test taxonomy (happy / edge / adversarial / composition). The transfer set *is* the operational definition of the behaviour being preserved, so its breadth bounds the safety of the conclusion. |

The caller (e.g. `semantic-compress`) owns deriving and confirming the transfer set; this capability consumes it. A thin transfer set yields a weak equivalence claim - the caller is responsible for flagging coverage, and the output records it.

Mechanism

For each case in the transfer set:

1. Run the runner (`references/runner-prompt.md`, the pure-wrapper prompt) once with `original` as the skill draft, on the case input, producing the **teacher transcript**. 2. Run the same runner once with `candidate` as the skill draft, on the **identical** case input, producing the **candidate transcript**. 3. Hand both transcripts to the **equivalence judge** (`references/equivalence-judge-prompt.md` - a focused compare-two-transcripts judge), which emits the per-case verdict and efficiency signal.

The runner and runner-prompt are the only execution primitive; the equivalence judge is the one comparison component, distinct from any absolute-quality lens. The judge compares observed behaviour to observed behaviour, never the candidate against what the original document *says* it should do. The full contract and schema are in `references/ab-equivalence.md`; the judge prompt and decision rule are in `references/equivalence-judge-prompt.md`.

Baseline caching (the teacher is captured once)

The `original` never changes across a multi-round transform loop, so its transcript per case is **captured once and reused across every round**. Only the candidate is re-run each round. This is a hard rule, not an optimization: re-running the teacher each round wastes runner budget and risks introducing teacher-side noise that the judge would mistake for a candidate change. The caller passes the cached teacher transcripts back in on rounds >= 2; this capability re-runs only the candidate. A budget ceiling on candidate re-runs belongs to the caller's loop, not here.

Verdict categories

Per case, the judge returns exactly one verdict:

| Verdict | Meaning | What it must cite | |---------|---------|-------------------| | `equivalent` | The candidate induced every behaviour and discipline the original induced. Incidental wording differences with no behavioural consequence are still `equivalent`. | Nothing required beyond the verdict. | | `candidate-regressed` | A behaviour or discipline the original induced is **absent** in the candidate. This is the failing verdict. | The **specific behaviour lost** - the discipline, step, or output the original produced and the candidate did not. | | `candidate-diverged`