/eval-creator

name: eval-creator
description: "[Beta] Creates permanent eval cases from promoted learnings and runs regression checks against them. Turns failures into test cases that prevent silent regression. This is the outer loop's regress-test step. Use when a learning is promoted and has a clear pass/fail condition, or on cadence to verify promoted rules still hold."

Eval Creator

Turns promoted learnings into permanent eval cases. Runs regression checks to verify promoted rules hold. This is the outer loop's **regress-test** step.

The blog says: "If a failure taught you something important, it should become a permanent test case. Otherwise the knowledge is still fragile."

When to Use

• **After harness-updater promotes a pattern** — create an eval for it
• **On cadence** — run all evals to check for regression
• **Before major releases** — verify the harness is holding
• **When a promoted rule seems to have stopped working** — diagnose with targeted eval run

Eval Directory Structure

.evals/
  EVAL_INDEX.md          # Index of all eval cases with status
  cases/
    eval-YYYYMMDD-001.md # Individual eval case
    eval-YYYYMMDD-002.md
    ...

Creating an Eval Case

Input

From harness-updater or manually:

• Pattern-Key of the promoted learning
• The rule that was added to the project instruction files (CLAUDE.md, AGENTS.md, .github/copilot-instructions.md)
• What to test (the assertion)
• Verification method

Eval Case Format

---
id: eval-YYYYMMDD-NNN
pattern-key: [from learning]
source: [LRN-YYYYMMDD-001, ERR-YYYYMMDD-003]
promoted-rule: "[the rule text in project instruction files]"
promoted-to: CLAUDE.md  # or AGENTS.md, .github/copilot-instructions.md, or equivalent
created: YYYY-MM-DD
last-run: YYYY-MM-DD
last-result: pass | fail | skip
---

## What This Tests

[One sentence: what failure this eval prevents from recurring]

## Precondition

[What must be true for this eval to be runnable]
- File X exists
- Project uses framework Y
- etc.

## Verification Method

[One of: grep-check, command-check, file-check, rule-check]

### grep-check
Search for a pattern that should (or should not) exist:

target: src/**/*.ts pattern: "hardcoded-secret-pattern" expect: not_found

### command-check
Run a command and check the exit code or output:

command: npm run typecheck expect_exit: 0

### file-check
Verify a file or section exists:

target: CLAUDE.md # or AGENTS.md, .github/copilot-instructions.md section: "## Verification" expect: exists

### rule-check
Verify a rule exists in an instruction file:

target: CLAUDE.md # or AGENTS.md, .github/copilot-instructions.md contains: "[the promoted rule text or key phrase]" expect: found

## Expected Result

**Pass:** [What "good" looks like]
**Fail:** [What regression looks like]

## Recovery Action

If this eval fails:
1. [Specific step to diagnose]
2. [Specific step to fix]
3. Re-run this eval to verify

Running Evals

Run All

Read `.evals/EVAL_INDEX.md`, iterate through all cases, execute each verification method.

Run by Pattern-Key

Filter to evals matching a specific pattern.

Run by Area

Filter to evals whose source files match an area (frontend, backend, etc.).

Execution

For each eval case:

1. **Check precondition** — if not met, mark as `skip` 2. **Execute verification method:**

• `grep-check`: Use Grep tool to search target files for the pattern
• `command-check`: Run the command via Bash, check exit code and/or output
• `file-check`: Use Read/Glob to verify file/section existence
• `rule-check`: Read the target file, search for the expected content
• `skill-check`: Run `quick_validate.py` on a skill directory (see Skill Validation below)
• `script-check`: Run a custom mcp-script by name (see Custom Verification Methods)

3. **Compare result** to expected 4. **Update `last-run` and `last-result`** in the eval case file 5. **Update `EVAL_INDEX.md`** with the result

Regression Report

## Eval Run: YYYY-MM-DD

**Total:** N evals
**Passed:** N
**Failed:** N
**Skipped:** N

### Failures

#### eval-YYYYMMDD-001 — [pattern-key]
- **What regressed:** [description]
- **Expected:** [X]
- **Got:** [Y]
- **Recovery action:** [from eval case]

### Summary
[All green / N regressions need attention]

Eval Index Format

`.evals/EVAL_INDEX.md`:

# Eval Index

| ID | Pattern-Key | Rule Summary | Last Run | Result | Created |
|----|-------------|-------------|----------|--------|---------|
| eval-YYYYMMDD-001 | auth-middleware-lock | Run migrations on test DB first | YYYY-MM-DD | pass | YYYY-MM-DD |
| eval-YYYYMMDD-002 | pnpm-not-npm | Use pnpm in this repo | YYYY-MM-DD | fail | YYYY-MM-DD |

Integration

Upstream

• **harness-updater** flags eval candidates after promoting a pattern (harness-updater is a plugin-bundle agent, `plugin/agents/harness-updater.md`; without the plugin, the human applying promotions manually flags eval candidates from the gap report instead)
• **learning-aggregator** identifies patterns with clear pass/fail conditions

Downstream

• Regression failures feed back into **self-improvement** as new error entries
• Persistent failures may indicate the promoted rule needs refinement → feed back to **harness-updater**

Scheduled Use

For projects with a CI pipeline, eval-creator can run as a scheduled check:

• Weekly: run all evals
• Per-PR: run evals related to changed files
• Post-promotion: run the newly created eval immediately

Custom Verification Methods (mcp-scripts)

Beyond the four built-in methods (grep-check, command-check, file-check, rule-check), projects can define custom verification tools as mcp-scripts for complex assertions that the built-ins can't express.

Example — an eval that verifies a promoted auth rule is enforced:

# In gh-aw workflow config
mcp-scripts:
  check-auth-middleware:
    lang: javascript
    description: "Verify all /admin r