/validate-evaluator

name: validate-evaluator
description: >
  Calibrate an LLM judge against human labels using data splits, TPR/TNR, and
  bias correction. Use after writing a judge prompt (write-judge-prompt) when you
  need to verify alignment before trusting its outputs. Do NOT use for code-based
  evaluators (those are deterministic; test with standard unit tests).

Validate Evaluator

Calibrate an LLM judge against human judgment.

Overview

1. Split human-labeled data into train (10-20%), dev (40-45%), test (40-45%) 2. Run judge on dev set and measure TPR/TNR 3. Iterate on the judge until TPR and TNR > 90% on dev set 4. Run once on held-out test set for final TPR/TNR 5. Apply bias correction formula to production data

Prerequisites

• A built LLM judge prompt (from write-judge-prompt)
• Human-labeled data: ~100 traces with binary Pass/Fail labels per failure mode
• Aim for ~50 Pass and ~50 Fail (balanced, even if real distribution is skewed)
• Labels must come from a domain expert, not outsourced annotators
• Candidate few-shot examples from your labeled data

Core Instructions

Step 1: Create Data Splits

Split human-labeled data into three disjoint sets:

| Split | Size | Purpose | Rules | |-------|------|---------|-------| | **Training** | 10-20% (~10-20 examples) | Source of few-shot examples for the judge prompt | Only clear-cut Pass and Fail cases. Used directly in the prompt. | | **Dev** | 40-45% (~40-45 examples) | Iterative evaluator refinement | Never include in the prompt. Evaluate against repeatedly. | | **Test** | 40-45% (~40-45 examples) | Final unbiased accuracy measurement | Do NOT look at during development. Used once at the end. |

Target: 30-50 examples of each class (Pass and Fail) across dev and test combined. Use balanced splits even if real-world prevalence is skewed — you need enough Fail examples to measure TNR reliably.

from sklearn.model_selection import train_test_split

# First split: separate test set
train_dev, test = train_test_split(
    labeled_data, test_size=0.4, stratify=labeled_data['label'], random_state=42
)
# Second split: separate training examples from dev set
train, dev = train_test_split(
    train_dev, test_size=0.75, stratify=train_dev['label'], random_state=42
)
# Result: ~15% train, ~45% dev, ~40% test

Step 2: Run Evaluator on Dev Set

Run the judge on every example in the dev set. Compare predictions to human labels.

Step 3: Measure TPR and TNR

**TPR (True Positive Rate):** When a human says Pass, how often does the judge also say Pass?

TPR = (judge says Pass AND human says Pass) / (human says Pass)

**TNR (True Negative Rate):** When a human says Fail, how often does the judge also say Fail?

TNR = (judge says Fail AND human says Fail) / (human says Fail)

from sklearn.metrics import confusion_matrix

tn, fp, fn, tp = confusion_matrix(human_labels, evaluator_labels,
                                   labels=['Fail', 'Pass']).ravel()
tpr = tp / (tp + fn)
tnr = tn / (tn + fp)

Use TPR/TNR, not Precision/Recall or raw accuracy. These two metrics directly map to the bias correction formula. Use Cohen's Kappa only for measuring agreement between two human annotators, not for judge-vs-ground-truth.

Step 4: Inspect Disagreements

Examine every case where the judge disagrees with human labels:

| Disagreement Type | Judge | Human | Fix | |-------------------|-------|-------|-----| | **False Pass** | Pass | Fail | Judge is too lenient. Strengthen Fail definitions or add edge-case examples. | | **False Fail** | Fail | Pass | Judge is too strict. Clarify Pass definitions or adjust examples. |

For each disagreement, determine whether to:

• Clarify wording in the judge prompt
• Swap or add few-shot examples from the training set
• Add explicit rules for the edge case
• Split the criterion into more specific sub-checks

Step 5: Iterate

Refine the judge prompt and re-run on the dev set. Repeat until TPR and TNR stabilize.

**Stopping criteria:**

• **Target:** TPR > 90% AND TNR > 90%
• **Minimum acceptable:** TPR > 80% AND TNR > 80%

**If alignment stalls:**

| Problem | Solution | |---------|---------| | TPR and TNR both low | Use a more capable LLM for the judge | | One metric low, one acceptable | Inspect disagreements for the low metric specifically | | Both plateau below target | Decompose the criterion into smaller, more atomic checks | | Consistently wrong on certain input types | Add targeted few-shot examples from training set | | Labels themselves seem inconsistent | Re-examine human labels; the rubric may need refinement |

Step 6: Final Measurement on Test Set

Run the judge **exactly once** on the held-out test set. Record final TPR and TNR.

Do not iterate after seeing test set results. Go back to step 4 with new dev data if needed.

Step 7 (Optional): Estimate True Success Rate (Rogan-Gladen Correction)

Raw judge scores on unlabeled production data are biased. If you need an accurate aggregate pass rate, correct for known judge errors:

theta_hat = (p_obs + TNR - 1) / (TPR + TNR - 1)

Where:

• `p_obs` = fraction of unlabeled traces the judge scored as Pass
• `TPR`, `TNR` = from test set measurement
• `theta_hat` = corrected estimate of true success rate

Clip to [0, 1]. Invalid when TPR + TNR - 1 is near 0 (judge is no better than random).

**Example:**

• Judge TPR = 0.92, TNR = 0.88
• 500 production traces: 400 scored Pass -> p_obs = 0.80
• theta_hat = (0.80 + 0.88 - 1) / (0.92 + 0.88 - 1) = 0.68 / 0.80 = **0.85**
• True success rate is ~85%, not the raw 80%

Step 8: Confidence Interval

Compute a bootstrap confidence interval. A point estimate alone is not enough.

import numpy as np

def bootstrap_ci(human_labels, eval_labels, p_obs, n_bootstrap=2000):
    """Bootstrap 95% CI for corrected success rate."""
    n = len(human_labels)
    estimates = []
    for _ in range(n_bootstrap):
        idx = np.random.choice(n, size=n,