/toolkit

name: toolkit
description: "Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md."
user-invocable: true
agent: toolkit-governance-engineer
allowed-tools:
  - Read
  - Write
  - Edit
  - Bash
  - Glob
  - Grep
  - Agent
  - Task
  - Skill
routing:
  force_route: true
  not_for: "application code (use workflow), code review (use review)"
  triggers:
    - "create skill"
    - "create agent"
    - "scaffold skill"
    - "scaffold agent"
    - "new skill"
    - "new agent"
    - "skill template"
    - "agent template"
    - "eval skill"
    - "evaluate agent"
    - "benchmark skill"
    - "benchmark agents"
    - "compare agents"
    - "A/B test agents"
    - "optimize description"
    - "evolve toolkit"
    - "toolkit evolution"
    - "self-improve"
    - "update routing tables"
    - "routing maintenance"
    - "generate claude.md"
    - "create claude.md"
    - "compose skills"
    - "bake-off"
    - "skill quality"
    - "weaker model"
    - "weak model uplift"
    - "improve skill for weaker models"
  category: meta-tooling
  pairs_with:
    - review
    - workflow

Toolkit

Ten modes covering the full toolkit lifecycle: creating, evaluating, and improving skills and agents; maintaining routing tables; generating CLAUDE.md; composing multi-skill DAGs; and running the evolution loop. Classify the request and follow the matching section.

Mode Selection

| Mode | Signals | Section | |------|---------|---------| | **Skill Creator** | create skill, scaffold skill, new skill, build a skill | Create Skill | | **Agent Creator** | create agent, scaffold agent, new agent | Create Agent | | **Skill Eval** | eval skill, benchmark skill, improve skill, bake-off | Evaluate Skill | | **Weak-Model Uplift** | weaker model, uplift skill, make a skill work for Opus 4.6, improve guidance from generated output | Uplift for Weaker Models | | **Agent Comparison** | compare agents, A/B test agents, benchmark agents | Compare Agents | | **Agent Evaluation** | evaluate agent quality, audit agent, grade agent | Evaluate Agent | | **Skill Composer** | compose skills, DAG orchestration, skill pipeline | Compose Skills | | **Routing Tables** | update routing tables, sync routing, routing drift | Update Routing | | **Toolkit Evolution** | evolve toolkit, self-improve, discover gaps | Evolve Toolkit | | **Generate CLAUDE.md** | generate claude.md, create claude.md, init | Generate CLAUDE.md |

---

Create Skill

Phases: **INTENT -> DRAFT -> TEST -> EVAL -> IMPROVE**

1. **Capture intent.** What should the skill do? When should it trigger? What output? Are outputs objectively verifiable (code, data) or subjective (writing, design)? 2. **Duplicate check.** Run `grep -i "<domain>" skills/*/SKILL.md` to check existing coverage. If an umbrella skill covers the domain, add a reference file instead. 3. **Write SKILL.md.** Follow `references/skill-creator/skill-template.md` for frontmatter structure. Apply Dense-Complete Writing standard. Frontmatter must include: name, description, routing (triggers, not_for, category, pairs_with), allowed-tools. 4. **Create test prompts.** 3 should-trigger, 2 should-not-trigger, 2 near-miss prompts. Save as `EVAL.md`. 5. **Run eval loop.** Execute test prompts with the skill loaded. Grade results. Iterate on the SKILL.md until eval passes. 6. **Register.** Run `python3 scripts/generate-skill-index.py` to update routing.

Load `references/skill-creator.md` for the full workflow. Deep references in `references/skill-creator/` cover progressive disclosure, artifact schemas, complexity tiers, error catalog, enrichment workflow, and more.

Scripts: `scripts/skill-creator/`

---

Create Agent

Phases: **DISCOVER -> DESIGN -> SCAFFOLD -> REGISTER -> VALIDATE**

1. **Discover.** Check for domain overlap: `grep -i "<domain>" agents/*.md`. If an existing agent covers the domain, add a `references/` file instead. 2. **Design.** Decide role type (reviewer/engineer/orchestrator), allowed tools, complexity, triggers (3-6 specific phrases), pairs_with (verify each exists), reference files, description (intent verb + domain + boundary clause), activation cases. 3. **Scaffold.** Write the agent file using `references/agent-creator/agent-frontmatter-template.md`. Follow `docs/PHILOSOPHY.md` for operator context structure. 4. **Register.** Run `python3 scripts/generate-agent-index.py`. 5. **Validate.** Run `python3 scripts/validate-references.py` to check reference file integrity. Test activation with the 3+2+2 prompt set.

Load `references/agent-creator.md` for full phases. Deep references in `references/agent-creator/` cover design patterns, frontmatter template, eval design.

---

Evaluate Skill

Three evaluation types: **trigger testing**, **A/B benchmark**, and **bake-off**.

1. **Trigger test.** Run each EVAL.md prompt. Grade: did the skill activate? Did it produce correct output? 2. **A/B benchmark.** Compare skill variants on the same prompts. Measure: accuracy, token usage, user satisfaction. Load `references/skill-eval/schemas.md` for grading schemas. 3. **Bake-off.** Head-to-head comparison of two skill variants. Load `references/skill-eval/bake-off-methodology.md`. 4. **Self-improve loop.** After eval, identify weaknesses, modify the SKILL.md, re-eval. Load `references/skill-eval/self-improve-loop.md`.

Load `references/skill-eval.md` for the full methodology.

---

Uplift for Weaker Models

Improve a skill, agent, or shared guide until a weaker model produces strong output with it. Load `references/weak-model-uplift.md` and follow its steps:

1. **Pick the target from data.** Query `~/.claude/learning/usage.db` and `learning.db` for heavily used or failing skills. 2. **Build tasks and checks first.** 4–8 tasks plus 1–2 held-out tasks; deterministic checks and a yes/no rubric written before any run. 3. **Run the arms.** No guidance and current guidance, two samples per task minimum, with `python3 scripts/weak_model_run.py`.