/gating

name: gating
description: >-
  Build and audit deterministic verification gates — a check that blocks a
  pipeline and can be shown to go red. Use when writing a calibration gate,
  CI check, validation script or pre-publication check for a numeric or
  empirical result; when a plausible-but-wrong value would survive review;
  when asking whether an existing test, linter rule or check could actually
  fail; and when a suite passes first try, passes suspiciously often, or was
  written by whatever produced the thing it checks. Triggers on "can this
  check fail", "known-bad", "negative control", "calibration gate", "sanity
  check my results", "is this test actually testing anything".
metadata:
  version: 0.3.0

gating

A gate is a check that blocks. Its only job is to go red when it should.

The characteristic failure is not a wrong check — a wrong check gets noticed. It is a check that **cannot fail**, which reports PASS forever and is indistinguishable from a working one from the outside. That is what makes this different from ordinary testing: the object under suspicion is the check.

When NOT to use this skill

Scope is ONE check and whether it can be made to fail.

| Situation | Use | |---|---| | Sequence several steps with branches and retries | flowing | | Run the repo's existing suite | run it | | Decide what to test at all | this skill has no opinion; that is design |

A gate is a thing that goes red. If nothing here can go red, there is no gate to audit.

The three obligations

Every gate owes these. A gate missing any of them is not yet a gate.

**1. An anchor outside your own code.** Something the check compares against that your implementation did not produce: a published constant, a closed-form answer, a conservation law, a degenerate case with a known result, an independent implementation. A check that compares this run to the last run only ever tells you the code still does what it did. See `references/anchors.md`.

**2. A known-bad it demonstrably rejects.** Break the subject the way it would plausibly break, run the gate, confirm red. Until you have done this you have not shown the gate works — you have shown it runs. This is the obligation people skip, because a passing gate feels like evidence.

Two things about known-bads that are easy to get wrong:

• **Validate it at the configuration it will run in.** A case tuned on a small

or fast setting can stop being bad at full size. An "untrained" grid built from one Lloyd iteration was genuinely zero-gain at m=2/K=16 and earned a real +0.10 dB at m=8/K=65536, where one iteration relocates ~63,000 empty cells toward the mode. It passed the fast gate and certified nothing about the real one. This matters more than it sounds, because `mutate.py` needs a fast gate variant and it is tempting to validate everything there.

• **Measure its *reach*.** One known-bad is the floor, not the goal. Name which

checks it exercises (`known_bad(..., covers=(...))`); the harness prints the checks no known-bad reaches. An audited gate had a single known-bad covering 1 of 8 checks — and the check its whole result rested on *accepted* the same bad case.

**3. A written statement of what it cannot catch.** Coverage holes are invisible from inside a green run: the gate is silent about the thing it does not look at, in exactly the same tone it uses for the thing it looked at and approved. The author has to assert the hole; nothing else will.

`scripts/gate.py` enforces obligations 2 and 3 mechanically — it returns exit code 2 (INCONCLUSIVE, not PASS) when a gate registers no known-bad or no coverage limit.

Building a gate

Work in this order. The first step is the one that determines whether the rest is worth anything.

**Name the wrong conclusion, not the component.** Not "check the quantizer is correct" but "prevent shipping *scalar wins at high bit rates* when that would really be an optimizer artifact." A gate aimed at a conclusion knows what counts as a near-miss; a gate aimed at a component just exercises the code.

**Find an anchor.** `references/anchors.md` lists the kinds, in rough order of strength, with the questions that find each one.

**Prefer brackets to point checks.** Assert a value lies strictly between two things it cannot legitimately pass: better than a baseline, worse than a theoretical bound. A one-sided check passes for a result that collapsed as readily as for one that is right — which is how an implementation that silently does nothing gets certified.

**Derive the tolerance from measured noise.** Run the thing several times, see how much it moves, put the threshold outside that. A tolerance picked for comfort tends to land wider than the defect you are trying to catch, and then it swallows it.

**Then check it is not too tight to mean anything.** The opposite failure is real and less obvious: a margin can be statistically impeccable and practically empty. A *paired* estimator — scoring both arms on one shared sample so the common fluctuation cancels — is the right way to measure a difference, and its standard error *shrinks as the two arms converge*. So "beats the baseline by 3 se" degenerates: a codebook perturbed by N(0, 1e-3) gained +0.0001 dB against a 3-se margin of 1.2e-06 and was accepted, while real ones gained 0.35–1.41 dB. The check certified *the effect is real*, not *the effect is worth having*. Those are different assertions and need different thresholds — and the second one has to come from an anchor (there, a published lattice codebook), never from the estimator, which knows nothing about what magnitude would matter.

**Build the known-bad and confirm red.** Then run `scripts/mutate.py` for the failures you did not think of.

**Wire it to a non-zero exit** and run it before the thing it gates, not after. A gate that runs after the results are written is a report.

Auditing an existing check suite

Given tests, a linter config, a CI job, or a gate someone already wr