/eval-creator

name: eval-creator
description: "[Beta] Creates permanent eval cases from promoted learnings and runs regression checks against them. Turns failures into test cases that prevent silent regression. This is the outer loop's regress-test step. Use when a learning is promoted and has a clear pass/fail condition, or on cadence to verify promoted rules still hold."

Eval Creator

Turns promoted learnings into permanent eval cases. Runs regression checks to verify promoted rules hold. This is the outer loop's **regress-test** step.

The blog says: "If a failure taught you something important, it should become a permanent test case. Otherwise the knowledge is still fragile."

When to Use

• **After harness-updater promotes a pattern** — create an eval for it
• **On cadence** — run all evals to check for regression
• **Before major releases** — verify the harness is holding
• **When a promoted rule seems to have stopped working** — diagnose with targeted eval run

Eval Directory Structure

.evals/
  EVAL_INDEX.md          # Index of all eval cases with status
  cases/
    eval-YYYYMMDD-001.md # Individual eval case
    eval-YYYYMMDD-002.md
    ...

Creating an Eval Case

Input

From harness-updater or manually:

• Pattern-Key of the promoted learning
• The rule that was added to the project instruction files (CLAUDE.md, AGENTS.md, .github/copilot-instructions.md)
• What to test (the assertion)
• Verification method

Eval Case Format

---
id: eval-YYYYMMDD-NNN
pattern-key: [from learning]
source: [LRN-YYYYMMDD-001, ERR-YYYYMMDD-003]
promoted-rule: "[the rule text in project instruction files]"
promoted-to: CLAUDE.md  # or AGENTS.md, .github/copilot-instructions.md, or equivalent
created: YYYY-MM-DD
last-run: YYYY-MM-DD
last-result: pass | fail | skip
evidence-level: presence | structural | behavioral
---

## What This Tests

[One sentence: what failure this eval prevents from recurring]

## Precondition

[What must be true for this eval to be runnable]
- File X exists
- Project uses framework Y
- etc.

## Verification Method

[One of: grep-check, command-check, file-check, rule-check, behavior-check]

### grep-check
Search for a pattern that should (or should not) exist:

target: src/**/*.ts pattern: "hardcoded-secret-pattern" expect: not_found

### command-check
Run a command and check the exit code or output:

command: npm run typecheck expect_exit: 0

### file-check
Verify a file or section exists:

target: CLAUDE.md # or AGENTS.md, .github/copilot-instructions.md section: "## Verification" expect: exists

### rule-check
Verify a rule exists in an instruction file:

target: CLAUDE.md # or AGENTS.md, .github/copilot-instructions.md contains: "[the promoted rule text or key phrase]" expect: found

`rule-check` proves only that guidance is installed. It must not be reported as
proof that an agent follows the guidance.

### behavior-check
Exercise the relevant behavior with synthetic positive and negative controls:

command: bash bench/run-contract-case.sh bounded-retry expect_exit: 0 positive_control: "transient failure is retried within budget" negative_control: "deterministic failure changes input or stops"

A behavioral pass requires observable evidence for both controls. Prefer the
repository's existing test, bench, or fixture mechanism; do not create a new
test framework for one eval.

## Expected Result

**Pass:** [What "good" looks like]
**Fail:** [What regression looks like]

## Recovery Action

If this eval fails:
1. [Specific step to diagnose]
2. [Specific step to fix]
3. Re-run this eval to verify

Running Evals

Run All

Read `.evals/EVAL_INDEX.md`, iterate through all cases, execute each verification method.

Run by Pattern-Key

Filter to evals matching a specific pattern.

Run by Area

Filter to evals whose source files match an area (frontend, backend, etc.).

Execution

For each eval case:

1. **Check precondition** — if not met, mark as `skip` 2. **Execute verification method:**

• `grep-check`: Use Grep tool to search target files for the pattern
• `command-check`: Run the command via Bash, check exit code and/or output
• `file-check`: Use Read/Glob to verify file/section existence
• `rule-check`: Read the target file, search for the expected content
• `behavior-check`: Run the existing synthetic fixture or bench case and

require the positive control to pass and the negative control to be rejected

• `skill-check`: Run `quick_validate.py` on a skill directory (see Skill Validation below)
• `script-check`: Run a custom mcp-script by name (see Custom Verification Methods)

3. **Compare result** to expected 4. **Check evidence strength** — a presence/structural method cannot satisfy a behavioral assertion. Mark the case `fail` with `insufficient_evidence` rather than upgrading the claim. 5. **Update `last-run` and `last-result`** in the eval case file 6. **Update `EVAL_INDEX.md`** with the result

Regression Report

## Eval Run: YYYY-MM-DD

**Total:** N evals
**Passed:** N
**Failed:** N
**Skipped:** N

### Failures

#### eval-YYYYMMDD-001 — [pattern-key]
- **What regressed:** [description]
- **Expected:** [X]
- **Got:** [Y]
- **Recovery action:** [from eval case]

### Summary
[All green / N regressions need attention]

Eval Index Format

`.evals/EVAL_INDEX.md`:

# Eval Index

| ID | Pattern-Key | Rule Summary | Last Run | Result | Created |
|----|-------------|-------------|----------|--------|---------|
| eval-YYYYMMDD-001 | auth-middleware-lock | Run migrations on test DB first | YYYY-MM-DD | pass | YYYY-MM-DD |
| eval-YYYYMMDD-002 | pnpm-not-npm | Use pnpm in this repo | YYYY-MM-DD | fail | YYYY-MM-DD |

Integration

Upstream

• **harness-updater** flags eval candidates after promoting a pattern (harness-updater is a plugin-bundle agent, `plugin/agents/harness-updater.md`; without the plugin, the huma