/output-eval-audit

name: output-eval-audit
description: Audit an existing eval suite for trustworthiness. Use when inheriting evals, suspecting evals miss real failures, or after significant pipeline changes.
allowed-tools: [Bash, Read]

Auditing an Eval Suite

Overview

Audit your eval suite to determine whether it actually catches real failures. This skill provides a structured diagnostic that identifies gaps in error analysis, evaluator design, judge validation, and dataset coverage, with concrete remediation steps for each finding.

When to Use

• Inheriting an eval suite from another team or developer
• Suspecting that evals pass but production quality is poor
• After switching models, rewriting prompts, or changing pipeline logic
• Periodic health check (quarterly or after major releases)

Step 1: Gather Artifacts

Read the eval infrastructure files for the workflow being audited:

src/workflows/<workflow_name>/
├── tests/
│   ├── datasets/           # YAML dataset files
│   │   ├── *.yml
│   │   └── ...
│   └── evals/
│       ├── evaluators.ts   # Evaluator definitions
│       ├── workflow.ts      # Eval workflow definition
│       └── *.prompt         # Judge prompt files

Inventory what exists:

| Artifact | File(s) | Count | |----------|---------|-------| | Evaluators | `tests/evals/evaluators.ts` | ? | | Eval workflow | `tests/evals/workflow.ts` | ? entries in `evals` array | | Judge prompts | `tests/evals/*.prompt` | ? | | Datasets | `tests/datasets/*.yml` | ? | | Datasets with ground_truth | ? of above | ? | | Datasets with last_output | ? of above | ? |

If any of these are missing entirely, note it and skip to "Starting From Zero" at the bottom.

Step 2: Run the Diagnostic

Evaluate each of the four areas below. For each, assign a status:

• **Pass** — Meets the standard
• **Warn** — Partially meets the standard, improvements needed
• **Fail** — Does not meet the standard, significant risk

---

Area 1: Error Analysis Grounding

**Question:** Were the evaluators derived from observed failure modes in real workflow traces?

**Check:**

• Do failure categories exist (documented in a file, comments, or commit history)?
• Does each evaluator map to a specific failure category?
• Or are evaluators measuring generic qualities ("quality score", "overall rating")?

**Pass criteria:**

• Each evaluator targets a named failure mode (e.g., "check_tone" targets tone mismatch, not "evaluate general quality")
• Failure categories were derived from reviewing real traces (not brainstormed)

**Common failures:**

• Evaluators named `evaluate_quality`, `check_overall`, `rate_output` — generic, not grounded in observed failures
• Evaluators were written based on what seemed important, not what actually fails
• No evidence of trace review before evaluator creation

**Remediation:** `output-eval-error-analysis` — Review 50+ traces and categorize actual failure modes before modifying evaluators

---

Area 2: Evaluator Design

**Question:** Are the evaluators well-designed for reliable automated evaluation?

**Check each evaluator in `tests/evals/evaluators.ts`:**

| Check | What to look for | |-------|------------------| | One failure mode per judge | Each `judgeVerdict()` evaluator targets exactly one criterion | | Binary verdicts | Judge prompts use pass/fail, not Likert scales (1-5) or multi-axis ratings | | Code-based where possible | Objective checks use `Verdict.*` helpers, not LLM judges | | Few-shot examples in judges | Judge `.prompt` files include pass, fail, and borderline examples | | Critique before verdict | Judge prompts request critique/reasoning before the verdict in structured output | | Appropriate criticality | `required` for blocking failures, `informational` for nice-to-have checks | | Correct interpret type | `interpret` config matches what the evaluator returns |

**Pass criteria:**

• All checks above are met for every evaluator

**Common failures:**

• A single judge prompt evaluates 3+ criteria simultaneously ("Rate tone, accuracy, and completeness")
• Judge prompts have no few-shot examples
• Deterministic checks (length, string contains, regex) use LLM judges instead of `Verdict.*`
• `interpret` type doesn't match evaluator return type (e.g., `judgeVerdict()` with `interpret: { type: 'boolean' }`)

**Remediation:** `output-eval-judge-prompt` — Redesign judge prompts following the four-component structure

---

Area 3: Judge Validation

**Question:** Have LLM judges been validated against human labels?

**Check for each LLM-based evaluator (those using `judgeVerdict()`, `judgeScore()`, `judgeLabel()`):**

| Check | What to look for | |-------|------------------| | Human labels exist | Datasets have `ground_truth.evals.<evaluator_name>.verdict` populated | | TPR/TNR measured | Validation results documented (file, comment, or commit) | | Train/dev/test split | Few-shot examples in the judge prompt come from a designated train split, not from the same data used for measurement | | Metrics meet threshold | TPR > 80% and TNR > 80% (target: > 90%) |

**Pass criteria:**

• Every LLM judge has documented TPR/TNR metrics above 80%
• Train/dev/test split was used (no data leakage)

**Common failures:**

• No validation at all — judges were written and immediately deployed
• Few-shot examples in the judge prompt are the same examples used to measure metrics (data leakage)
• "It seems to work" without quantitative measurement
• Only raw accuracy reported (masks class imbalance)

**Remediation:** `output-eval-validate-judge` — Calibrate each judge against human labels using TPR/TNR

---

Area 4: Dataset Coverage

**Question:** Do the datasets adequately cover the failure space?

**Check:**

| Check | What to look for | |-------|------------------| | Dataset count | Minimum 10 for simple workflows, 20+ for complex ones | | Diversity | Datasets vary across multiple input dimensions, not just happy paths | | Failure representation | At least 30% of datasets have