/output-eval-judge-prompt

name: output-eval-judge-prompt
description: Design effective LLM judge .prompt files for evaluators. Use when creating judgeVerdict/judgeScore/judgeLabel prompts, or when existing judges produce unreliable results.
allowed-tools: [Read, Write, Edit]

Designing LLM Judge Prompts

Overview

An LLM judge evaluates workflow output for a **single, specific failure mode** identified during error analysis. This skill covers how to design the `.prompt` file that powers `judgeVerdict()`, `judgeScore()`, or `judgeLabel()` calls. For the file format basics, see `output-dev-prompt-file`. For error analysis, see `output-eval-error-analysis`.

Prerequisites

Before writing a judge prompt:

1. **Error analysis is complete** — You have identified the specific failure mode this judge targets (from `output-eval-error-analysis`) 2. **20+ labeled examples** — At least 20 pass and 20 fail traces for this failure mode, with `ground_truth` labels in dataset YAML files 3. **Code-based check ruled out** — Confirmed that `Verdict.*` helpers (contains, matches, gte, etc.) cannot reliably detect this failure

The Four Components

Every effective judge prompt has exactly four components.

1. Task and Criterion

State the single failure mode being evaluated. Be specific and observable.

**Good criteria (specific, observable):**

• "Does the blog post maintain a formal tone throughout, or does it slip into casual language?"
• "Does the output contain any URLs that are fabricated rather than drawn from the input?"
• "Does the summary faithfully represent the source material without adding claims not present in the original?"

**Bad criteria (vague, holistic):**

• "Is this output high quality?"
• "Rate the overall effectiveness of this response"
• "How good is this content?"

2. Pass/Fail Definitions

Define exactly what constitutes pass and fail. **Always binary** — no Likert scales, no 1-5 ratings, no "partially meets criteria."

PASS: The blog post uses formal language throughout. Professional vocabulary,
complete sentences, no slang, no contractions, no first-person casual asides.

FAIL: The blog post contains one or more instances of casual language: slang,
contractions ("don't", "can't"), informal asides ("pretty cool", "super important"),
or conversational filler ("honestly", "basically").

Why binary: Likert scales create ambiguous boundaries (what's the difference between a 3 and a 4?). Binary forces precise definitions that LLMs can apply consistently and that you can validate against human labels.

3. Few-Shot Examples

Include at least three labeled examples: one clear pass, one clear fail, and one borderline case. **Borderline examples are the most valuable** — they teach the judge where the decision boundary lies.

Draw examples from your **training split only** (see `output-eval-validate-judge`). Never use dev or test examples as few-shot — that's data leakage.

Each example must include:

• The relevant input/output excerpt
• A detailed critique explaining the reasoning
• The verdict (pass or fail)

4. Structured Output

Request JSON output with **critique before verdict**. This forces the judge to reason before deciding, which improves accuracy.

{
  "critique": "Detailed analysis of the output against the criterion...",
  "verdict": "pass"
}

Always put `critique` first in the schema. If `verdict` comes first, the judge commits to a decision before reasoning.

Full `.prompt` File Example

A judge for the "tone mismatch" failure mode:

# tests/evals/judge_tone@v1.prompt
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-haiku-4-5-20251001
temperature: 0
maxOutputTokens: 1500
---

<system>
You are an evaluation judge. Your task is to determine whether a blog post maintains the requested tone throughout.

## Criterion

Assess whether the blog post consistently uses the requested tone. A single paragraph that breaks tone is a failure.

## Definitions

PASS: The blog post maintains the requested tone in every paragraph. Word choice, sentence structure, and rhetorical style all align with the requested tone.

FAIL: The blog post contains one or more paragraphs where the tone shifts away from what was requested. Common failures include:
- Formal request but casual language appears ("pretty cool", "super important", contractions)
- Professional request but opinionated editorializing appears
- Technical request but oversimplified explanations appear

## Examples

### Example 1: PASS
Requested tone: formal
Blog excerpt: "The implications of quantum computing for cryptographic security are substantial. Current encryption standards rely on the computational infeasibility of factoring large prime numbers, a guarantee that quantum algorithms may undermine."
Critique: The excerpt uses professional vocabulary ("implications", "computational infeasibility"), complete sentences, no contractions, and maintains an academic register. Consistent formal tone throughout.
Verdict: pass

### Example 2: FAIL
Requested tone: formal
Blog excerpt: "Quantum computing is basically going to break all our encryption. It's pretty wild when you think about it — everything we thought was secure might not be."
Critique: The excerpt contains multiple casual markers: "basically", "pretty wild", contractions ("It's", "might not be"), and conversational filler ("when you think about it"). This directly violates the formal tone request.
Verdict: fail

### Example 3: BORDERLINE (fail)
Requested tone: formal
Blog excerpt: "Quantum computing represents a paradigm shift in computational capability. The technology is incredibly promising, though it's important to note the current limitations in qubit stability and error correction."
Critique: Mostly formal, but contains "incredibly promising" (informal intensifier) and "it's" (contraction). While the overall register is professional, these lapses break the formal tone requirement