/agentsop-regression-gate

name: agentsop-regression-gate
version: 0.1.0
phase: D
tier: core
frequency: high
status: opinionated
description: Build a held-out eval set, run it on every prompt/model change, and block regressions in CI. An LM change is a code change — gate it with a test suite (eval set + metric + threshold). Cross-framework SOP not surfaced by any single base skill.

regression-gate — Eval Set + Metric + Threshold, Wired Into CI

> "Every subsequent change must be gated on these numbers." > — Synthesized from [[llamaindex]] Stage 2 (eval loop *before* optimizing) [llamaindex-sop-skill/SKILL.md:114-126]

> "Compiled program beats baseline on a *held-out* test set (not the val set used in optimization)." > — [[dspy]] Stage 3 exit criterion [dspy-sop-skill/SKILL.md:101]

This is an **enhancement overlay**. The regression-gate SOP exists only as fragments scattered across base skills — [[llamaindex]] `OP-10 EvalLoop` ("gate every change"), [[dspy]] train/dev/test split + metric — and is never assembled as a standalone cross-framework discipline. It is the discipline that turns a one-off eval into a *gate*: a test suite that runs in CI on every prompt/model/retriever change and fails the build on regression. It consumes a metric from `[[agentsop-metric-design]]` and, for domain-specific held-out sets, hands off to `[[agentsop-domain-eval-set]]`.

---

1. 何时激活 (When to Activate)

Activate when **any** of these is true:

• **Any prompt change you want to ship safely**: a prompt edit, a system-message tweak, a few-shot-demo swap is about to merge and you have no automated way to know if it made things worse.
• **Any model change**: swapping GPT-4o → a cheaper/newer model, a temperature change, a provider migration. An LM change silently shifts the whole output distribution. [[dspy]] Case B: "If you optimize a complex pipeline for GPT-4, it usually breaks on a smaller model" [dspy-sop-skill/SKILL.md:184].
• **Any retriever/chunking/reranker change in a RAG pipeline**: every such change needs a quantitative gate. [[llamaindex]] `OP-10`: "Quantitative regression test for every chunking / embedding / retriever / prompt change" [llamaindex-sop-skill/SKILL.md:235].
• **Recurring "it got worse" surprises**: the team keeps shipping changes that users report as regressions after the fact. The fix is a gate, not more careful review.
• **Setting up CI for an LLM app** and there is no eval job in the pipeline.

**Do NOT activate for:**

• **One-shot tasks** with no production surface — there is nothing to regress. ([[dspy]] boundary: "Summarize this email once → raw API call" [dspy-sop-skill/SKILL.md:259].)
• **The signature/task is still changing daily** — gate only after the I/O contract stabilizes, else you re-baseline every commit. ([[dspy]] boundary [dspy-sop-skill/SKILL.md:260].)
• **No willingness to define any success criterion** — without a metric there is nothing to gate. Route to `[[agentsop-metric-design]]` first; if the user refuses, this skill cannot help.

---

2. 核心心智模型 (Core Mental Model)

**"An LM change is a code change. Gate it with a test suite: eval set + metric + threshold."**

You already gate code with unit tests in CI: a change that breaks a test fails the build. A prompt edit, a model swap, a chunk-size tweak are *also* changes to the system's behavior — but they slip through review because their effect is statistical, not a stack trace. The regression gate is the missing unit test for LM behavior.

The gate is exactly three artifacts plus a wiring step:

   eval set        metric           threshold          CI wiring
  (held-out QA)  (ex,pred)->score   (fail if <X / drop>Y)  (block merge)
        │              │                  │                    │
        └──────────────┴──────────────────┴────────────────────┘
                          REGRESSION GATE

Three load-bearing principles:

1. **The eval set is held out and frozen.** It is a labelled, version-controlled fixture that the prompt/model under test has *never seen*. [[dspy]] is explicit: the compiled program must beat baseline on a held-out test set "not the val set used in optimization" [dspy-sop-skill/SKILL.md:101]. The split is `train / dev / test`; the **gate runs on test only**. If the eval set leaks into the prompt (few-shot demos, instructions), the gate measures memorization, not quality.

2. **The metric comes from `[[agentsop-metric-design]]`, not invented here.** This skill does not design metrics — it consumes one. A bad metric makes the gate theatre: it will pass changes that hurt users and block changes that help them. The metric must be human-calibrated *before* it gates anything ([[agentsop-metric-design]] `OP-M05`).

3. **The threshold is a policy, not a number you guess.** Two common shapes: an **absolute floor** (fail if score < X) and a **relative no-regression** (fail if score drops > Y from the committed baseline). Relative is the regression gate proper; absolute is a quality bar. Most teams use both: a floor for "never ship below this," plus a no-regression delta for "this PR must not make it worse."

Build the eval loop *before* you optimize anything

[[llamaindex]] Stage 2 is named "Build the eval loop **before** optimizing anything" [llamaindex-sop-skill/SKILL.md:114]. The anti-pattern it names is A3: "No eval loop; debug by anecdote" [llamaindex-sop-skill/SKILL.md:348]. The gate is the institutional form of that loop — once it exists, every change is debugged by number, not by vibe.

---

3. SOP (Standard Operating Procedure)

0. Confirm activation (§1); confirm a metric exists or invoke [[agentsop-metric-design]]
1. BUILD eval set:  generate candidates -> curate to a golden set -> freeze + version
2. SPLIT:           train / dev / test; the GATE runs on TEST only
3. PICK metric:     consume from [[agentsop-metric-design]] (do not invent here)
4. SET threshold:   absolute floor AND/OR relative no-regression delta
5. WIRE into CI:    run eval on every prompt/model/