/02-metric-design

name: metric-design
description: >
  Use when the user has evaluation principles or a dataset but needs help choosing
  the right graders, designing evaluation metrics, creating LLM-as-judge prompts,
  combining multiple metrics into a composite score, or building an automated
  evaluation pipeline. Also use when the user mentions grader selection, metric
  design, judge prompt engineering, rubric design, evaluation pipeline code,
  or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.

Metric Design

Select, configure, and combine evaluation graders into a working pipeline. You choose the right tool for each evaluation dimension — from zero-cost code checks to LLM judges — and produce executable `GradingRunner` code that runs on OpenJudge.

> **Requires OpenJudge** (`pip install py-openjudge`). This skill is intentionally > SDK-centric — grader selection, `GradingRunner`, and aggregators are OpenJudge APIs. The > design/decision logic still applies if you use another harness; only the code does not.

When to Activate

• User has eval dimensions/principles but doesn't know which grader type to use
• User wants to write an LLM-as-judge prompt for a specific failure mode
• User needs a composite score combining multiple evaluation dimensions
• User wants to auto-generate graders from labeled data instead of writing them manually
• User's current evaluation is all LLM-based and too expensive/too slow

Checklist

You MUST create a task for each item and complete them in order:

1. **Select grader types** — per dimension, pick the right grader class 2. **Create custom graders** — write judge prompts (4-component) or function graders 3. **Auto-generate if applicable** — use OpenJudge Generator for cold starts 4. **Run anti-pattern scan** — check for Likert, missing few-shot, vague criteria 5. **Build pipeline code** — assemble GradingRunner with graders + aggregators

Step 1: Select Grader Type Per Dimension

For each evaluation dimension, walk this decision tree (first match wins):

1. Can a deterministic rule check this?
   → StringMatchGrader / JsonValidatorGrader / FunctionGrader (zero cost, 100% consistent)
   Examples: exact match for classification labels, regex for format checks,
             JSON schema validation, keyword presence/absence

2. Does it require semantic understanding of text quality?
   → LLMGrader with built-in class (low cost, pre-optimized)
   Examples: CorrectnessGrader (factual match), RelevanceGrader (on-topic check),
             HallucinationGrader (faithfulness to context)

3. Does it involve agent behavior (tool calls, planning, memory)?
   → Agent-specific LLMGrader
   Examples: ToolSelectionGrader, TrajectoryAccuracyGrader, MemoryAccuracyGrader

4. Does it involve code execution or syntax?
   → CodeExecutionGrader / SyntaxCheckGrader
   Examples: test case pass rate, syntax validity, code style checks

5. Does it require external tool calls to verify (web search, database lookup)?
   → AgenticGrader (expensive, use only when necessary)
   Examples: fact-checking against live sources, cross-referencing databases

Grader Selection Cheat Sheet

| Output type | Recommended grader | Cost | |------------|-------------------|------| | Classification label | `StringMatchGrader` | Free | | JSON structure | `JsonValidatorGrader` + `JsonMatchGrader` | Free | | Free text correctness | `CorrectnessGrader` | LLM call | | Factual accuracy (grounded) | `HallucinationGrader` | LLM call | | Response relevance | `RelevanceGrader` | LLM call | | Instruction following | `InstructionFollowingGrader` | LLM call | | Tool call selection | `ToolSelectionGrader` | LLM call | | Agent trajectory | `TrajectoryAccuracyGrader` | LLM call | | Code correctness | `CodeExecutionGrader` | Free | | Custom quality check | Custom `LLMGrader` | LLM call | | External fact verification | `AgenticGrader` | LLM + tool calls |

**Why this order matters**: Every LLM-based grader adds cost, latency, and non-determinism. A `StringMatchGrader` costs nothing and always gives the same answer. Exhaust deterministic options before reaching for an LLM judge.

Step 2: Create Custom Graders

LLMGrader: The Four-Component Template

When no built-in grader fits, create a custom `LLMGrader`. Every judge prompt needs exactly these four components (adapted from community best practice):

**Component 1 — Task & Criterion**: What this judge evaluates. One thing only.

You are evaluating whether a customer support response correctly identifies
and uses the customer's order number from the conversation context.

**Component 2 — Binary Pass/Fail Definitions**: Concrete, observable conditions.

PASS: The response references the correct order number exactly as it appears
in the context. If multiple orders exist, the response addresses the right one.

FAIL: The response uses a wrong order number, omits the order number when one
was provided, or references an order not present in the context.

Why binary and not Likert? Because two human annotators agree on "pass vs fail" far more often than on "3 vs 4 out of 5." Binary forces a clear decision boundary. If you need severity levels, use multiple binary judges (e.g., "factually wrong" + "dangerously wrong").

**Component 3 — Few-Shot Examples**: At minimum 1 pass, 1 fail, 1 borderline. The borderline example is the most valuable — it teaches the judge where the boundary is.

Example 1 (PASS):
Context: "Order #12345: shipped May 10"
Response: "Your order #12345 was shipped on May 10 and arrives May 12."
Critique: The response uses the exact order number (#12345) and matches the
ship date from context. No fabrication or omission.
Result: Pass

Example 2 (FAIL):
Context: "Order #12345: shipped May 10"
Response: "Your order #12346 is on its way!"
Critique: The response uses order #12346 but the context only mentions #12345.
This is a fabricated order number, not a typo — #12346 doesn't exist.
Resu