/spec-driven-eval

name: spec-driven-eval
description: Scores how completely an implementation fulfills a PRD/spec, case by case, and produces a single comparable final grade. Invoke only when explicitly named (e.g. run spec-driven-eval); do not auto-trigger. Use when benchmarking spec-driven implementations, grading acceptance criteria, evaluating whether a feature was 100% implemented, comparing multiple implementations of the same PRD, or auditing implementation and test coverage (unit and e2e) against product requirements. Do NOT use for planning or building features (use tlc-spec-driven), writing PRDs, or general code review unrelated to a spec.
license: CC-BY-4.0
disable-model-invocation: true
metadata:
  author: Waldemar Neto - github.com/waldemarnt
  version: '1.0.0'

Spec-Driven Implementation Evaluation

Evaluate a spec-driven-development (SDD) effort against a PRD **case by case** (acceptance criterion by acceptance criterion), scoring **implementation** and **tests** separately, and roll up to a single comparable final grade. Designed for benchmarking: the same PRD evaluated across different SDD frameworks — and the same effort evaluated twice — must yield comparable, reproducible numbers.

Two subjects, two questions

This eval answers two independent questions, and keeps their verdicts separate because they fail independently:

1. **How good is the framework at respecting and extracting requirements?** — Did it honor the PRD and stay in bounds (*respect*: `Final` implementation side + Scope `S`), and did it surface the implicit requirements the PRD only implied, without noise (*extract*: Elicitation `E`)? 2. **How good is the harness at ensuring they are all implemented?** — Does the test suite prove every sanctioned requirement is actually built (`T` + Engineering Gates `G`)?

> **The linkage that makes "all implemented" well-defined:** the harness is held accountable for the **full sanctioned requirement set = `PRD acceptance criteria ∪ valid E-additions`**. A valid requirement the framework extracted but the harness never tests is a *harness* miss (`T`), not a framework miss (`E` keeps the extraction credit). Extraction defines the verification target.

Scoring is **checklist-based**: every criterion is decomposed into atomic binary (MET / UNMET) checks, each backed by `file:line` evidence. Binary decomposition is the design choice that makes the grade reproducible — graded/Likert scales (`is this a 3 or a 4?`) are the dominant source of evaluator-to-evaluator disagreement; binary checks raise inter-evaluator agreement to roughly human level. Partial credit is **derived** from the fraction of checks met, never judged on a sliding scale.

When to use

• "Evaluate/score this PRD case by case and give a final grade"
• "Was this story/feature implemented 100%?"
• Benchmarking multiple spec-driven implementations of the same PRD
• Auditing implementation **and** test coverage (unit + e2e) against acceptance criteria

Inputs required

1. **The PRD** (ground-truth product intent) — the user story acceptance criteria are the unit of evaluation. 2. **The implementation** (production code). 3. **The tests** (unit + e2e). 4. **The SDD-derived artifacts** — `spec.md` and `tasks.md` (refined ACs, requirement IDs, derived requirements). Required for the Elicitation `E` and Scope `S` axes (they grade *these* against the PRD). When absent, `Final`/`T` still run, but report `E`/`S` as `n/a — no derived spec`. The **PRD remains the source of truth** for what counts as "expected"; the derived spec is what gets graded for respect and extraction.

**Scoping the diff.** Before scoring, use git to identify which files changed for this implementation. That **diff surface** is the primary search scope for all `file:line` evidence in steps 4 and 8:

git diff <base>..<head> --name-only   # branch or PR
git diff --name-only HEAD             # uncommitted changes
git status --short                    # include untracked new files

Record the diff surface in the report. Evidence outside it is still valid (e.g. a pre-existing file was modified), but note when a check relies on files not in the diff — that may indicate the wrong base was chosen.

---

Quick start

New to running this end-to-end? See [quickstart.md](references/quickstart.md) for the 4-chat-session flow (freeze baseline → plan → implement → evaluate) with paste-ready prompts. Read it when you need the operational how-to; the rest of this file is the scoring methodology.

---

Core rules (read first)

These five rules govern every score. They exist to make the grade auditable and reproducible.

1. **Evidence or zero.** Every MET check MUST cite evidence as `file:line` (or `file:startLine-endLine`). No located evidence ⇒ the check is UNMET. Never award credit from assumptions or from the PRD restating intent. 2. **Search-before-zero (anti false-negative).** Before marking a check UNMET for "not found", record the search actually performed — starting with the diff-surface files from the scoping step, then the grep/glob terms tried and any additional files/dirs inspected. A check scored UNMET must confirm the behavior is absent from the diff surface, not just from an ad-hoc grep across the full repo. UNMET means *searched and genuinely absent*, not *did not look*. If no search is shown, the check is **not yet scored**, not UNMET. 3. **Read the path end-to-end — including the data shape.** A check is MET only if you traced the real code/test path, not because a symbol name matches. For an emitted/returned/persisted artifact, **inspect the constructed payload object itself**, not just the call site — a present `emit(...)`/`return ...` does not prove the named field is in the payload (see the Conjunction rule). A test that *exercises* a behavior but does not *assert* it does **not** meet a verification check. 4. **Judge ≠ author for benchmarks.** When grading to compare implementations, the evaluating model should differ from