/gen-evals

description: "Generate EVAL-*.md test cases for an agent from its prompt. Usage: /gen-evals <agent-name> [--count N]"
argument-hint: "<agent-name> — e.g. architect, qa-engineer, security-officer"
user-invocable: true
allowed-tools: Read, Write, Bash, Glob, Grep
model: haiku

<!-- great_cto-managed -->

You are the great_cto `/gen-evals` command. You generate synthetic evaluation test cases for a named agent, saving them in the established `tests/eval/EVAL-*.md` format. This is the Sprint 2 implementation of the Hermes self-evolution pattern: **synthetic eval dataset generation from agent documentation**.

Step 1 — Parse arguments and locate agent

AGENT_NAME="${ARGUMENTS%% *}"  # first word
COUNT=$(echo "$ARGUMENTS" | grep -oE '\-\-count [0-9]+' | grep -oE '[0-9]+' || echo 20)
[ -z "$AGENT_NAME" ] && echo "Usage: /gen-evals <agent-name> [--count N]" && exit 1

# Locate agent file in plugin dir or repo
PLUGIN_DIR=$(ls -d ~/.claude/plugins/cache/local/great_cto/*/ 2>/dev/null | sort -V | tail -1 | sed 's|/$||')
AGENT_FILE="${PLUGIN_DIR}/agents/${AGENT_NAME}.md"
[ ! -f "$AGENT_FILE" ] && AGENT_FILE="agents/${AGENT_NAME}.md"
[ ! -f "$AGENT_FILE" ] && echo "ERROR: agent not found: ${AGENT_NAME}" && exit 1

echo "Generating $COUNT eval cases for: $AGENT_NAME"
echo "Reading: $AGENT_FILE"

Step 2 — Read agent definition

Read the full agent file. Focus on:

• What the agent is supposed to do (description, step-by-step instructions)
• What it must NOT do (constraints, rejection criteria, anti-patterns)
• What evidence / artifacts it must produce (output format, required fields)
• The domain it covers (archetype, compliance area, specialisation)

This is the source material for generating realistic test cases.

Step 3 — Generate test cases (LLM synthesis)

Based on the agent definition, generate $COUNT realistic, adversarial test cases.

**For each test case, produce:**

| # | Input / Scenario | Expected behaviour | Pass criterion |

**Generation rules (from Hermes PLAN.md pattern):**

• Split 60% routine / 40% adversarial (edge cases, subtle failure modes)
• Routine: valid inputs where the agent should succeed clearly
• Adversarial: inputs designed to expose the most common failure modes for this agent type
• Expected behaviour: rubric-based, not exact-match (e.g. "flags at least 2 injection vectors" not "outputs the word SQL")
• Pass criterion: objectively verifiable — a judge model can determine pass/fail

**Common adversarial shapes by agent type:**

• `*-reviewer`: prompt injection in input, scope creep, false positive suppression, missing high-severity finding
• `architect`: over-engineering, ignoring constraints from PROJECT.md, circular dependency in plan
• `qa-engineer`: accepting flaky tests, missing edge case category, wrong severity rating
• `security-officer`: missing critical (P0) finding, accepting ambiguous evidence as proof
• `senior-dev`: touching files outside owned-files list, breaking existing tests, re-deriving decided choices
• `pm`: decomposing incorrectly, missing dependency, wrong agent assignment

Generate cases in structured tables followed by a pass threshold. Threshold = 80% default (16/20), adjust down to 70% for adversarial-heavy sets.

**Split the cases into tuning + holdout (SIA `data/public` vs `data/private`):**

• ~70% of cases → `## Cases (tuning)` — visible to ai-prompt-architect for iteration.
• ~30% of cases (min 3) → `## Holdout cases` — gate-only, used by `scripts/eval-gate.mjs`

to block prompt revisions that regress. Put the **hardest adversarial cases** in holdout — they are the most valuable overfit detector and must stay unseen during prompt tuning.

Step 4 — Write EVAL file(s)

Group the $COUNT cases into EVAL files of max 5-8 cases each (matching the existing `tests/eval/` convention). Each file covers one specific failure mode or scenario cluster.

**File naming:** `tests/eval/EVAL-<agent-name>-<slug>.md` where `<slug>` is a 2-4 word kebab description of the scenario cluster.

**File format (MUST match existing tests/eval/ convention exactly):**

# EVAL-<agent-name>-<slug>.md

> Agent: <agent-name> · Generated by /gen-evals on <YYYY-MM-DD>

## Scenario
<2-3 sentences describing what this cluster tests and why it matters>

## Cases (tuning)
| # | Scenario | Expected | Pass |
|---|---|---|---|
| 1 | <input description> | <expected behaviour — rubric-based> | <pass criterion> |
...

## Holdout cases
| # | Scenario | Expected | Pass |
|---|---|---|---|
| H1 | <hardest adversarial input — kept unseen> | <expected behaviour> | <pass criterion> |
...

## Pass threshold
<N>/<total>. (applies to each split)

## Run
`node tests/eval/runner.mjs --filter EVAL-<agent-name>-<slug>`
`node tests/eval/runner.mjs --filter EVAL-<agent-name>-<slug> --split holdout`  # gate evidence

## Cross-refs
- Agent: <agent-name> · Shape: <A-F from continuous-learner shapes>

## History
| Date | Version | Result | Notes |
|---|---|---|---|

Step 5 — Emit baseline run command

After writing all EVAL files:

EVAL_FILES=$(ls tests/eval/EVAL-${AGENT_NAME}-*.md 2>/dev/null | wc -l | tr -d ' ')
echo ""
echo "Generated $EVAL_FILES EVAL files for $AGENT_NAME"
echo ""
echo "Run baseline:"
echo "  export ANTHROPIC_API_KEY=sk-ant-..."
echo "  node tests/eval/runner.mjs --filter EVAL-${AGENT_NAME}"
echo ""
echo "Baseline scores will be the 'before' line in future /crystallize propose PRs."

Output rules

• Generate realistic test inputs — not toy examples. An LLM judge must be able to

evaluate them without the full project context.

• Do NOT include private project names, real credentials, or PII in test scenarios.
• Each case must be independently evaluable (no dependency on other cases).
• Keep `Expected` column concise (≤ 15 words) — the judge gets full context separately.
• If the agent has a specific output format (e.g. structured findings), test that format

is honoured in ≥2 cases.