/06-prompt-regression

name: prompt-regression
description: >
  Use when the user has changed a prompt (system prompt, RAG template, agent instruction,
  etc.) and wants to know whether the candidate is better or worse than the baseline.
  Also use when the user mentions prompt A/B testing, prompt comparison, prompt
  optimization validation, "did my prompt change help," or prompt regression testing.
  Outputs per-dimension win rates with statistical significance using OpenJudge
  PairwiseAnalyzer.

<HARD-GATE> NO conclusion about which prompt is better WITHOUT bootstrap 95% CI reported. NO candidate declared "better" WITHOUT position-debiased (swap-aggregate) comparison. NO comparison with fewer than 10 samples per axis — CI is too wide to be meaningful. </HARD-GATE>

Prompt Regression

Compare two prompts head-to-head and determine, with statistical rigor, whether the candidate is better, worse, or tied on each evaluation dimension.

When to Activate

• You changed the system prompt and want to verify it's actually better
• You're iterating on RAG answer templates
• You're optimizing agent step-by-step instructions
• You want data to support a prompt change decision

Checklist

You MUST create a task for each item and complete them in order:

1. **Load and analyze prompts** — diff the baseline vs candidate 2. **Derive comparison dimensions** — from the prompt changes + task type 3. **Select graders per dimension** — pairwise, judge, or rule 4. **Run position-debiased comparison** — swap-aggregate to eliminate order bias 5. **Compute statistics** — win rates + bootstrap 95% CI per dimension 6. **Present results** — per-dimension verdict with confidence intervals

Fast path: run the bundled script

Don't hand-write the win-rate + bootstrap math (the swap-aggregation and CI are easy to get wrong). Run the bundled, tested script (`scripts/pairwise.py`, standard library only, **no OpenJudge dependency**):

python scripts/pairwise.py --comparisons comparisons.jsonl --candidate candidate --baseline baseline

Each comparison row: `{"id","model_a","model_b","score","dimension"?}` where `score >= 0.5` means `model_a` won. Emit two rows per query with A/B **swapped** to debias position. The script reports per-dimension candidate/baseline/tie rates, bootstrap 95% CI, and a verdict (`BETTER` / `WORSE` / `TIED` / `INSUFFICIENT_EVIDENCE` / `INCONCLUSIVE`; exit 0 only if better). `--self-test` to verify it.

Steps below explain how to derive dimensions and produce the comparisons (with OpenJudge or any judge); the inline snippets are the reference behind the script.

Step 1: Load and Analyze Prompts

Read the baseline and candidate prompts. Identify:

• **Task type**: chatbot / RAG generation / code review / translation / summarization

/ agent instruction / other

• **What changed**: added constraints, changed tone, new examples, different output

format, expanded/shortened instructions

• **Intent of change**: what problem was the user trying to fix?

Step 2: Derive Comparison Dimensions

Based on the task type and what changed, derive 3-5 comparison dimensions.

Dimension templates by task type

**Chatbot / Conversational**:

• Answer relevance — does it address the user's question?
• Tone appropriateness — does the tone match context?
• Factual accuracy — no fabricated information
• Conciseness — doesn't ramble or over-explain
• Instruction following — obeys system prompt constraints

**RAG Generation**:

• Faithfulness — grounded in retrieved documents
• Citation accuracy — correctly references sources
• Completeness — covers all aspects of the query
• No hallucination — no claims beyond documents

**Code Review / Generation**:

• Bug detection — finds real issues
• False positive rate — doesn't flag correct code
• Actionability — suggestions are specific and implementable
• Code style — follows conventions

**Agent Instructions**:

• Tool selection — picks the right tool
• Step efficiency — minimal steps to goal
• Error recovery — handles failures gracefully
• Output format — follows specified structure

Each dimension gets:

• An `id` (slug)
• A one-sentence description
• A grader type: `pairwise` or `judge` or `rule`

Step 3: Select Graders

Decision priority: 1. **Can a rule check this?** → `FunctionGrader` or `StringMatchGrader`. Free, deterministic. Example: output length, keyword presence, JSON validity. 2. **Is there a reference answer?** → `pairwise` against reference. 3. **Subjective quality, no reference?** → `pairwise` A/B comparison. 4. **Single-output judgment needed?** → `judge` (binary pass/fail per output).

Step 4: Run Position-Debiased Comparison

Pairwise comparison with swap-aggregate

LLM judges have position bias — the first response shown wins 5-15% more often. Swap-aggregate eliminates this: run each comparison twice with swapped positions, keep only consistent wins:

from openjudge.graders.llm_grader import LLMGrader
from openjudge.graders.schema import GraderMode
from openjudge.runner.grading_runner import GradingRunner
from openjudge.analyzer.pairwise_analyzer import PairwiseAnalyzer

# Judge prompt for relevance comparison
relevance_judge = LLMGrader(
    model=model,
    name="relevance_compare",
    mode=GraderMode.POINTWISE,
    template="""
Compare Response A and Response B for the query below.
Which response better addresses the user's question?

Query: {query}
Response A: {response_a}
Response B: {response_b}

Score 1.0 if A is better, 0.0 if B is better, 0.5 if tied.
Respond in JSON: {{"score": <float>, "reason": "<explanation>"}}
""",
)

# Build pairwise dataset with position swap
dataset = []
for sample in test_samples:
    # Original order
    dataset.append({
        "query": sample["query"],
        "response_a": baseline_outputs[sample["id"]],
        "response_b": candidate_outputs[sample["id"]],
        "metadata": {"model_a": "baseline", "model_b": "candidate"},
    })
    # Swapped order — critical for debiasing