/agentsop-metric-design

name: agentsop-metric-design
version: 0.1.0
phase: D
tier: core
frequency: high
status: opinionated
description: >-
  Decomposed, multi-criteria metric design for LLM pipelines. The metric IS the model —
  change the metric and the optimizer changes behavior. Decompose by default; bool during
  compile, float during eval; calibrate against human; mitigate judge bias. Search keywords:
  LLM-as-judge, llm as judge, eval metric, evaluation score, scoring function, rubric,
  RAGAS, G-Eval, judge bias, verbosity bias, how to evaluate LLM output.

metric-design — Decomposed, Multi-Criteria Metrics for LLM Pipelines

> "It's unproductive to launch optimization runs using a poorly designed program or a bad metric." > — DSPy core team [dspy.ai/learn/optimization/overview/]

> "LLM judges exhibit self-preference, recency, rubric-order, score-ID, and length biases." > — Synthesized from [arxiv.org/pdf/2506.02592, arxiv.org/pdf/2509.26072]

This is a **tool skill**. It produces a metric function (and a calibration receipt) that other skills consume — DSPy compilers (MIPROv2 / GEPA / BootstrapFewShot), LlamaIndex `FaithfulnessEvaluator`/`RelevancyEvaluator`/`RetrieverEvaluator`, LangGraph eval judges, RAGAS, TruLens. The metric is the optimization target. Get it wrong and every downstream optimizer is theatre.

---

1. 何时激活 (When to Activate)

Activate when **any** of these is true:

• **Optimization runs**: a DSPy / OpenAI-Evals / RAGAS / TruLens job is about to consume a `metric(example, pred) -> bool|float`. The metric drives gradient-free search; bias propagates into the artifact.
• **RAG evaluation**: deciding chunk size, reranker, hybrid alpha, retriever-k. A bad metric here picks the wrong chunking strategy and you ship it.
• **Agent benchmarks**: tool-use, multi-step, planning. A single holistic LLM judge cannot distinguish "wrong tool" from "right tool, wrong args".
• **Prompt tuning that has gone past 2 manual iterations**: if you're tuning a prompt and have no quantitative metric, you are guessing. Stop the prompt edits, write the metric.
• **Production regression test**: every chunking / embedding / retriever / prompt PR change needs a metric gate (LlamaIndex `OP-10 EvalLoop`).

**Do NOT activate for:**

• One-off exploratory prompt tests where no decision rides on the output.
• Tasks where exact-match / unit-test / schema-validity already gives ≥95% of signal — don't over-engineer.
• Where the user explicitly refuses to commit to any evaluation criteria (then `dspy-sop` will refuse to compile anyway; this skill cannot help).

---

2. 核心心智模型 (Core Mental Model)

**"The metric IS the model. Change the metric, change the behavior."**

A DSPy/GEPA/MIPRO optimizer is a black-box search that maximizes `metric(pred, example)`. Whatever the metric rewards, the compiled prompt will produce. If the metric prefers verbose, hedged answers (which an LLM judge will, by default — judges over-prefer length [arxiv.org/pdf/2506.02592]), the optimizer will produce verbose, hedged answers. **A bad metric beats a good optimizer every time.**

Three corollaries:

1. **Decompose by default.** A single holistic LLM-as-judge call ("is this answer good? rate 1-5") collapses orthogonal axes (factuality, tone, length, relevance) into one noisy scalar. Decompose into N orthogonal yes/no sub-judges, then aggregate. Same number of LM calls in the limit, vastly less noise. 2. **Bool during compile, float during eval.** Same metric function, two return types based on the `trace` argument. Compile-time `bool` prevents the optimizer from chasing noise in the middle of the distribution; eval-time `float` gives gradient for reporting and debugging. [dspy.ai/learn/evaluation/metrics/] 3. **The metric must be calibrated against humans.** ≥20 spot-checks where you (the human) rate the same examples the metric does. If metric disagrees with human on >20% of cases, **fix the metric before any compile**. Otherwise the optimizer just learns the metric's bias.

Why holistic judges fail (the catalog)

| Bias | What it does | Source | |---|---|---| | Length bias | Judges prefer longer answers regardless of quality | [arxiv.org/pdf/2506.02592] | | Self-preference | A judge from family X prefers outputs from family X | [arxiv.org/pdf/2506.02592] | | Recency / position | Last option in a pairwise rated higher | [arxiv.org/pdf/2509.26072] | | Rubric-order | Criteria listed first weighted more | [arxiv.org/pdf/2509.26072] | | Score-ID | "5" and "10" anchor differently across rubrics | [arxiv.org/pdf/2509.26072] | | Provenance | Knowing the source model biases the rating | [arxiv.org/pdf/2506.02592] |

Full catalog in `references/R2-judge-bias-catalog.md`.

---

3. SOP (Standard Operating Procedure)

0. Confirm activation criteria (§1)
1. DECOMPOSE: list orthogonal criteria
2. CHOICE: bool vs float per criterion
3. WRITE: implement sub-judges + aggregator
4. BIAS-TEST: probe for length, position, self-preference
5. CALIBRATE: ≥20 human spot-checks; agreement ≥80%
6. SHIP: hand metric to optimizer / eval loop

Stage 1 — Decompose

Write down the criteria the user *actually* cares about. For an open-ended Q&A:

• **Factual?** (yes/no) — answer makes no unsupported claims
• **On-topic?** (yes/no) — addresses the question
• **Concise?** (length penalty, scalar) — token count vs budget
• **Non-hedging?** (yes/no) — no "as an AI language model..." or "it depends" cop-outs
• **Cites source?** (yes/no) — if grounded retrieval is required

For RAG specifically, copy LlamaIndex's triad: **Faithfulness** (answer entailed by context), **Relevancy** (answer addresses query), **Retriever quality** (MRR / hit-rate on labeled QA pairs). See `references/R1-source-evidence.md`.

**Output of this stage**: a checklist of 3–6 sub-criteria, each with type (`bool` / scalar) and aggregation rule (AND for hard gates, weighted sum for soft).

Stage 2 — Bool vs Float Choice (per criterion)

Per [dspy.ai/learn/evaluation/metr