/skill-autobench

name: skill-autobench
version: 1.0.0
description: |
  Author an eval for an existing skill from its REAL usage history, not its
  spec. Mine invocations from the brain's conversation archive
  (conversations/) and per-harness session transcripts — a user correction
  after an invocation is the gold signal — then synthesize an eval_contract
  plus 4-8 replayable cases with honesty labels (SPEC-DERIVED vs
  HISTORY-IMPLIED) and stage the result at skills/<name>/eval/autobench-<date>.md
  as PENDING-HUMAN-APPROVAL. Never rewrites SKILL.md. Ships two guard
  companions: panel integrity (multi-model judging must prove each provider
  actually responded) and the fail-improve taxonomy (logged LLM-fallback
  cases convert to deterministic code over time).
triggers:
  - "skill autobench"
  - "autobench"
  - "write the eval from usage history"
  - "synthesize an eval for this skill"
  - "mine how this skill is actually used"
  - "build a benchmark from my corrections"
  - "verify the eval panel"
  - "did all providers return"
requires:
  - dir:conversations/
mutating: true
writes_pages: false
writes_to:
  - skills/<name>/eval/
upstream: skill-autobench@fc834ee + panel-integrity@fc834ee + fail-improve-loop@fc834ee (taxonomy only)

skill-autobench — write the eval from lived usage

> **Convention:** see [conventions/brain-first.md](../conventions/brain-first.md) — > mining starts in the brain. Search the conversation archive before touching > raw transcript files, and never declare "no history" without having queried > the brain first. > > **Convention:** see [conventions/model-routing.md](../conventions/model-routing.md) — > mining and synthesis run on the cheap tier by default. The full multi-model > judging pass is an explicit opt-in (see Contract).

The self-improving loop has three legs: an **eval**, a **variant generator** (SkillOpt), and a **replay + judge harness** (`gbrain eval cross-modal`). The generator and the judge ship with gbrain. The persistently missing leg is the **eval author** — someone has to WRITE the eval, and a spec-derived benchmark only tests what the skill promised, not what users actually asked for or what actually went wrong. This skill writes the eval from reality instead of imagination.

Pipeline

1. MINE — extract real invocation windows

Substrates, in priority order:

1. **Brain conversation archive** — pages under `conversations/`, populated by the conversation-archive skill (hard dependency for this substrate: if it hasn't ingested your history yet, run it first). Search for the target skill's name, trigger phrases, and output shapes:

   gbrain search "<skill-name>"
   gbrain query "when did I use <skill-name> and what did I ask for"

2. **Per-harness session transcripts, as available** — use what the harness exposes; do not assume a layout. `gbrain transcripts recent --full` reads the configured local transcript corpus (local-only by design). Claude Code keeps per-project session JSONL under `~/.claude/projects/`; other harnesses have their own session stores. Absent stores are simply skipped.

From each hit, extract an **invocation window**: the user ask before the invocation, the invocation turn itself, and the 2 turns after — because that is where corrections live. **A user correction after an invocation is the gold signal**: it is a real, observed failure mode, and it becomes a `hard_fail` plus a replayable case.

**FAIL-CLOSED:** if no substrate yields a single real invocation of the target skill, emit an honest no-history report (substrates checked, queries run, windows scanned, zero matches) and stop. Do NOT invent "typical" invocations. For a skill with no history, the right tool is `gbrain skillopt <name> --bootstrap-from-skill` (spec-derived, and honest about it) — see Dedup.

2. SYNTH — turn windows into a proposed eval

From the mined windows plus the current SKILL.md, produce:

• A proposed **eval_contract**: goal, dimensions, hard_fails. Dimensions come

from observed asks; hard_fails encode observed corrections.

• **4-8 replayable cases**, each shaped

`{input, expected_behavior, failure_mode_to_catch}` — realistic input, a checkable expected behavior, and the named failure mode the case exists to catch.

• **Spec-vs-usage gaps**: "the spec says X, users consistently ask Y."

**HONESTY LABELS are mandatory.** Every dimension and every case is labeled:

• `HISTORY-IMPLIED` — a real mined window backs it; cite which one.
• `SPEC-DERIVED` — inferred from SKILL.md only; no usage evidence.

Never conflate the two. If history is thin or off-target, say so prominently at the top of the staged file ("GROUNDING WARNING: only N windows found, none exercised the core path") instead of padding with fabricated evidence.

**Privacy scrub before staging:** staged evals live in the skill repo and are distributable. Mined windows contain real names, companies, and deals — rewrite every case onto placeholder slugs (`alice-example`, `acme-example`) before writing the file. A history-grounded case keeps its shape and failure mode, never its real entities.

3. STAGE — human gate, always

Write `skills/<name>/eval/autobench-<date>.md` with frontmatter `status: PENDING-HUMAN-APPROVAL`.

**This skill NEVER rewrites SKILL.md** — not the eval_contract, not the body, not the triggers. Merging the staged eval is the human's decision. (This is a workflow contract the agent must honor, not a mechanically-enforced gate.)

The loop (after approval)

1. Human reviews, edits, and approves the staged eval; the approved eval_contract is merged into the skill's frontmatter explicitly. 2. Convert approved cases into `skills/<name>/skillopt-benchmark.jsonl` lines and run `gbrain skillopt <name>` — this is the SkillOpt surface extension: a history-grounded benchmark replacing the spec-derived bootstrap. 3. Judge outputs through the native gate:

   gbrain eval cross-modal --task "<what the outpu