/skill-eval

name: skill-eval
description: "Evaluate skills: trigger testing, A/B benchmarks, structure validation, head-to-head bake-offs."
user-invocable: false
argument-hint: "<skill-name>"
allowed-tools:
  - Read
  - Write
  - Bash
  - Grep
  - Glob
  - Agent
routing:
  triggers:
    - improve skill
    - test skill
    - eval skill
    - benchmark skill
    - skill triggers
    - skill quality
    - self-improve skill
    - skill self-improvement
    - improve skill with variants
    - bake-off
    - bake off
    - head-to-head
    - head to head
    - compare implementations
    - grade two versions
    - which skill is better
  not_for: "creating a new skill from scratch (use skill-creator). This skill evaluates and benchmarks skills that already exist. NOT: static structural/standards-compliance grading of agents (use agent-evaluation)"
  pairs_with:
    - agent-evaluation
    - verification-before-completion
  complexity: Medium-Complex
  category: meta

Skill Evaluation & Improvement

Measure and improve skill quality through empirical testing — because structure doesn't guarantee behavior, and measurement beats assumption. Also covers head-to-head bake-offs of two peer implementations of the same artifact (Mode F).

Reference Loading Table

| Signal | Load These Files | Why | |---|---|---| | reading or writing eval artifacts: evals.json, grading.json, metrics.json, history.json | `schemas.md` | Loads detailed guidance from `schemas.md`. | | improving a skill via variant generation and blind A/B promotion | `self-improve-loop.md` | Loads detailed guidance from `self-improve-loop.md`. | | "bake-off", "head-to-head", "compare implementations", "grade two versions", "which persona skill is better" | `bake-off-methodology.md` | Loads the bake-off rubric, anti-rationalization gate, fold-filter, and worked persona example. |

Instructions

Phase 1: ASSESS — Determine what to evaluate

**Step 1: Identify the skill**

# Validate skill structure first
python3 -m scripts.skill_eval.quick_validate <path/to/skill>

This checks: SKILL.md exists, valid frontmatter, required fields (name, description), kebab-case naming, description under 1024 chars, no angle brackets.

**Step 2: Choose evaluation mode based on user intent**

| Intent | Mode | Script | |--------|------|--------| | "Test if description triggers correctly" | Trigger eval | `run_eval.py` | | "Optimize/improve the description through autoresearch" | Route to `agent-comparison` | `optimize_loop.py` | | "Compare skill vs no-skill output" | Output benchmark | Manual + `aggregate_benchmark.py` | | "Validate skill structure" | Quick validate | `quick_validate.py` | | "Self-improve skill" / "optimize skill" / "improve skill with A/B" | Self-improvement loop | `references/self-improve-loop.md` | | "Bake-off" / "head-to-head grade these two" / "compare X vs Y implementation" | Head-to-head bake-off | `references/bake-off-methodology.md` |

**GATE**: Skill path confirmed, mode selected.

Phase 2: EVALUATE — Run the appropriate evaluation

Mode A: Trigger Evaluation

Test whether a skill's description causes Claude to invoke it for the right queries.

**Step 1: Create eval set** (or use existing)

Create a JSON file with 8-20 test queries. **Eval set quality matters** — use realistic prompts with detail (file paths, context, casual phrasing), not abstract one-liners. Focus on edge cases where the skill competes with adjacent skills.

Example of good eval queries:

[
  {"query": "ok so my boss sent me this xlsx file (Q4 sales final FINAL v2.xlsx) and she wants profit margin as a percentage", "should_trigger": true},
  {"query": "Format this data", "should_trigger": false}
]

**Why**: Real users write detailed, specific prompts. Abstract queries don't test real triggering behavior. Overfitting descriptions to abstract test cases bloats the description and fails on real usage.

**Step 2: Run evaluation**

python3 -m scripts.skill_eval.run_eval \
  --eval-set evals.json \
  --skill-path <path/to/skill> \
  --runs-per-query 3 \
  --verbose

This spawns `claude -p` for each query, checking whether it invokes the skill. Runs each query 3 times for reliability. Output includes pass/fail per query with trigger rates. Default 30s timeout; increase with `--timeout 60` if needed for complex queries.

**Constraints applied**:

• Always run baseline eval before making improvements
• 3 runs per query ensures statistical reliability
• Verbose output shows per-query pass/fail during eval runs

**GATE**: Eval results available. Proceed to improvement if failures found.

Mode B: Description Optimization

Automated loop that tests, improves, and re-tests descriptions using Claude with extended thinking.

python3 -m scripts.skill_eval.run_loop \
  --eval-set evals.json \
  --skill-path <path/to/skill> \
  --max-iterations 5 \
  --verbose

This will: 1. Split eval set 60/40 train/test (stratified by should_trigger) — prevents overfitting to test cases 2. Evaluate current description on all queries (3 runs each for reliability) 3. Use `claude -p` to propose improvements based on training failures 4. Re-evaluate the new description 5. Repeat until all pass or max iterations reached 6. Select best description by **test** score (not train score — prevents overfitting) 7. Open an HTML report in the browser

**Why 60/40 split**: Improvements should help across many prompts, not just test cases. Training on failures, validating on holdout ensures generalization.

**Why report HTML**: Visual reports enable quick review of which queries improved, which regressed, and what the new description looks like.

**GATE**: Loop complete. Best description identified.

Mode C: Output Benchmark

Compare skill quality by running prompts with and without the skill.

**Step 1: Create test prompts** — 2-3 realistic user prompts

**Step 2: Run with-skill and without-skill** in parallel subagents:

For each test prom