/optimizing-skills

name: optimizing-skills
description: Disciplined, validation-gated revision of an EXISTING skill so each edit is a measured improvement rather than a guess. Use when editing, revising, or tuning a skill that already exists and there is evidence it underperforms (observed failures, drift, complaints) — invoke by name, or have versioning-skills / creating-skill defer to it before applying edits. Not for authoring a brand-new skill from scratch (use creating-skill) or one-off prose.
metadata:
  version: 0.3.0

Optimizing Skills

Treat the skill document as the parameter under optimization: change it only when the change **demonstrably beats the version you already ship**. This is the discipline distilled from SkillOpt (microsoft/SkillOpt, arXiv:2605.23904) — its training apparatus dropped, its reproducibility discipline kept. The point is to stop editing skills on intuition and start editing them on evidence.

Core principle

A skill edit is only worth shipping if it strictly improves measured behavior on a held-out check. Most edits that *feel* like improvements don't move the needle, and some quietly regress. The gate below is what separates a real improvement from a confident guess.

The gate — run it every revision

1. **Assemble a held-out check set.** 3–8 representative tasks/prompts the skill should handle well, and it **must include the failure(s) that prompted this revision**. Keep the set fixed across the revision so before/after scores are comparable. 2. **Hold two versions.** `best` = what you currently ship (never let it silently degrade). `candidate` = `best` + your proposed edits. 3. **Score both on the check set.** "Run" here = dispatch each check task to the **Agent tool** (`subagent_type=general-purpose`) with the skill version in context, or evaluate by hand for small sets. Give the scoring agent the skill version and the task, and nothing else. Never hand it the ledger, your revision notes, or the diagnosis that motivated the edit: it will solve the task from those instead of from the skill, and the score stops measuring the skill. WikiSkill (arXiv:2608.27454) ablated exactly this and lost 2.8 points of final quality, 7.8 on their hardest split, by letting the worker read the improver's knowledge store. Score **per criterion**, not one collapsed pass/fail. When a task carries several criteria, the criterion that decides accept/reject is **the failure that prompted this revision**; the others are regression guards that must not get worse. Collapsing criteria masks the win: in the down-skilling-v1.2.0 retro, the edit drove architectural hallucination 60%→0% while an unrelated length criterion stayed 0/5 in both arms — a single combined pass/fail scored that as a 0–0 tie and would have rejected a large, real improvement. 4. **Accept only if `candidate` strictly beats `best`** on the triggering-failure criterion, with no regression guard worse. Ties → reject, keep `best`. An edit that doesn't move the needle does not ship.

**When the skill's own output is compiled by an Agent** (down-skilling and creating-skill produce a prompt an author writes from the SKILL), score **≥2 author samples per version**, or fix one author across both arms. A single author sample per arm lets author capability dominate the edit effect: the same down-skilling edit measured 95%→0% with one author pair and 60%→0% with another — real either way, but n=1 cannot tell a real edit from a lucky author.

This two-tier `best`/`candidate` split is the heart of it: a working revision can explore, but the shipped skill only ever ratchets upward.

Bounded edits (the "textual learning rate")

Cap edits per revision — **default ~4** distinct add/replace/delete operations, fewer as the skill matures. Large speculative rewrites drift and destroy your ability to attribute a regression to a cause. If a revision wants more edits than the budget, rank and keep the top ones (below) and let the rest wait.

Reflect: failures first, then successes

Separate the evidence before proposing edits:

• **Failure reflection.** Across the failing cases, find the *common,

systematic* pattern — not a one-off edge case. Propose edits that fix the pattern. **Failures take priority** in any merge.

• **Success reflection.** Across cases that already work, find generalizable

patterns worth encoding so they survive future edits. Reinforce; don't duplicate.

For both: edits must **generalize** (never hardcode task-specific values), and must **not duplicate** content already in the skill — patch genuine gaps only.

Rank when over budget

When candidate edits exceed the budget, keep them in this priority order:

1. **Systematic impact** — fixes a recurring failure across many cases, not one. 2. **Complementarity** — fills a real gap rather than restating existing content. 3. **Generality** — phrased as a durable principle, not tied to one task/entity. 4. **Actionability** — concrete, followable guidance over vague advice.

Drop the rest. They can return next revision if still warranted.

Protect the hard-won core

If a skill has a battle-tested core that routine edits keep eroding, fence it off and treat it as off-limits to fast edits. Revisit it only on a **deliberate longitudinal review**: compare the *same* check tasks across several versions to catch slow drift and regressions that single-edit review misses. (SkillOpt fences this region with HTML-comment markers and only rewrites it at epoch boundaries — the same idea, manual cadence.)

Recording rejected proposals

**FIRST action of any revision, before you score anything:**

python3 scripts/skill_ledger.py check --skill <name> --before best.md --after candidate.md

Exit 1 means this exact edit was proposed before and failed the gate. Read the recorded criterion and score, then propose something else. Do not re-score it.

**After the gate decides, record the outcome — acce