/evolve-skill

name: evolve-skill
description: Run the full skill evolution pipeline -- harvest sessions, discover signals, build golden dataset, eval baseline, evolve via DSPy, compare scores
argument-hint: "<skill-name> [--optimizer gepa|mipro|bootstrap]"

Evolve a skill via DSPy optimization

Run the complete skill evolution pipeline for a single skill. Harvests fresh session data, discovers new negative patterns, builds a golden eval dataset, scores the baseline, runs DSPy optimization, and presents a before/after comparison for review.

Arguments

SKILL_NAME=$1  (required: e.g., "code-review", "pinescript", "planning")
OPTIMIZER=$2   (optional: "gepa" (default), "mipro", or "bootstrap")

Parse from: `$ARGUMENTS`

If no skill name provided, ask the user which skill to evolve. Show skills with the most harvested data as suggestions.

Pipeline

Maximize parallelism. Steps within the same group run concurrently (use background subagents or parallel bash). Steps across groups are sequential.

Group A (parallel): Harvest + Discover

Run these two concurrently:

**Step 1: Harvest sessions (full, all projects)**

python3 distillery/scripts/distiller.py harvest-sessions

Report: total examples harvested, how many attributed to the target skill.

**Step 2: Discover new negative signal patterns**

python3 distillery/scripts/distiller.py discover-signals --top 20

Present the top candidates to the user. If any look like genuine dissatisfaction patterns (not neutral task requests), ask whether to add them to `_NEGATIVE_SIGNAL_PATTERNS` in `distiller.py` before proceeding. If patterns are added, re-run harvest (Step 1) to update signal classifications.

If no new patterns worth adding, continue.

Group B (sequential): Build golden

Depends on Group A completing.

**Step 3: Build golden eval dataset**

RECOMMENDED (human-label) path — post-2026-07-07 harvest data is mostly `ambiguous` (no typed user outcome), and a golden set dominated by `ambiguous` drives GEPA to degenerate results:

python3 distillery/scripts/distiller.py build-golden <skill> --top 20
# → writes candidates.jsonl. Open it, set each "label" to positive / negative / skip
#   (drop the ambiguous ones as "skip" unless you can grade them), then:
python3 distillery/scripts/distiller.py approve-golden <skill>

`approve-golden` writes `label` into `signal` for every kept row and HARD-ERRORS on any unknown label (including a left-over `ambiguous`), so the golden set is fully graded before GEPA runs.

Fast path (only when the harvested signal is already well-graded — mostly positive/negative, few ambiguous):

python3 distillery/scripts/distiller.py build-golden <skill> --top 20 --auto

`--auto` labels straight from harvested signal and prints a stderr WARNING when >50% of the rows are `ambiguous`; if you see that warning, stop and switch to the human-label path above.

Report: examples selected, positive/negative split, mean quality score.

Group C (parallel): Eval baseline + Evolve

Run these two concurrently -- both read from the golden dataset, neither writes to the other's output.

**Step 4: Eval baseline**

Run the eval judge as **in-session sub-agents** (no billed `claude -p`):

python3 distillery/scripts/distiller.py dspy-eval <skill> --dataset golden --emit-tasks

This returns `{count, tasks:[{index, prompt, ...}]}`. Dispatch one sub-agent per task (Agent tool, batched ~8); each returns its judge JSON. Collect `[{index, signal, session_id, skill_version, response}]`, then aggregate + record history:

python3 distillery/scripts/distiller.py dspy-eval <skill> --dataset golden --score-from-verdicts @<file>

Record the baseline composite score and per-dimension scores. This is the "before" measurement.

**Step 5: Evolve**

python3 distillery/scripts/distiller.py evolve <skill> --optimizer <optimizer> --iterations 5 --save

If the optimizer produces changes:

• Show the diff
• Report growth percentage and constraint pass/fail
• If constraints fail (>20% growth or >15KB), note the violation

If no changes produced, report that the baseline is already Pareto-optimal for this metric and suggest trying a different optimizer or improving the golden dataset.

Group D (sequential): Eval evolved + Review

Depends on Group C completing.

**Step 6: Eval evolved (if changed)**

If Step 5 produced an evolved skill and it was saved, score the **evolved text**, not the live skill. The `--emit-tasks` call MUST carry `--skill-file` pointing at the saved candidate, or the judge re-measures the baseline and the "Evolved" column is a copy of "Baseline":

python3 distillery/scripts/distiller.py dspy-eval <skill> --dataset golden --emit-tasks \
  --skill-file distillery/.eval-data/<skill>/evolved-SKILL.md

Then dispatch the judge sub-agents and aggregate exactly as in Step 4 (`--score-from-verdicts @<file>` — this step takes no `--skill-file`; the override only changes the prompt built at emit-tasks time). The example set is identical to Step 4 because relevance still keys off the live skill's keywords, so the two composites are directly comparable.

Note: any past comparison run WITHOUT `--skill-file` measured the baseline twice; its "delta" is noise. Re-run those before trusting them.

Present a comparison table:

| Metric         | Baseline | Evolved | Delta    |
|----------------|----------|---------|----------|
| Composite      | 0.64     | 0.71    | +0.07 (+11%) |
| Correctness    | 7.0      | 7.8     | +0.8     |
| Procedure      | 5.0      | 5.5     | +0.5     |
| Conciseness    | 6.8      | 7.2     | +0.4     |

**Step 7: Review and apply**

Present the user with: 1. The diff from Step 5 2. The score comparison from Step 6 (or note if no changes) 3. Constraint status (growth %, size)

Ask: "Apply the evolved skill to `plugins/whetstone/skills/<skill>/SKILL.md`?"

If approved:

• Copy the evolved text to the skill's SKILL.m