/skill-eval

name: skill-eval
description: 'Measure whether a skill helps a named task or needs revision or removal. Use when: a bounded routing or coding evaluation is requested; conformance alone cannot show benefit.'
practices:
- measurement-over-assertion
- ab-testing
skill_api_version: 1
hexagonal_role: supporting
consumes:
- skill-source-package
produces:
- probe-package
- probe-result.v1
context_rel:
- kind: supplier-to
  with: skill-builder
user-invocable: true
metadata:
  tier: meta
  dependencies: []
  capabilities: ["author_seeded_probe","run_probe_tier","evaluate_skill_decision"]
  effects: ["write_probe_package","dispatch_probe_producer"]
  canonical_status: canonical
  disposition: keep_specialist
  stability: experimental

/skill-eval

Answer one named maintenance decision: **retain, revise, remove, or insufficient evidence**. Choose the measurement that can answer that decision, use the caller's accepted cases and resource envelope, make one scoped recommendation, and stop. A completed evaluation does not require a positive difference.

This is an optional specialist. The repository's selected runner owns execution and bounds; native results own measurements; BD and Git retain their authority. Do not add a core skill, AO evaluation command, scheduler, dashboard, second tracker, or mandatory review merely to run an experiment.

Choose the question

| Caller decision | Measurement | What it can establish | |---|---|---| | Does loading this skill change a specific observable act? | Behavioral probe with `scripts/probe-skill.sh` | Behavior change on that scenario; not correct code or productivity | | Does this package or version improve engineering outcomes at acceptable cost? | Repository-selected controlled coding comparison, such as `evals/skills-rpi` | Endpoint outcomes and cost on selected tasks; independent completion only when required exact-subject evidence exists | | Does a qualified memory update help later work? | Separate frozen-versus-updated memory transfer test | Narrow later-task reuse evidence with skill and runtime held fixed | | What happened in ordinary runs? | Existing native accounting and acceptance evidence | Observational failures, repairs and cost; not causal skill benefit |

Start from the caller's intended decision, not a mandatory quiz. Reuse an existing accepted decision and scope. For a behavioral question, name one observable action (a file written, tool used, criterion rejected); a belief such as “understands validation” needs translation into an action. For coding or memory questions, name unchanged task acceptance and the maintenance choice.

Procedure

1. **Fix the decision and bounds.** Name the subject package/version or qualified memory update, relevant cases, allowed runtime and existing aggregate time, trial and cost limits. Do not infer billing enforcement from token counters. Smoke runs, infrastructure retries, interrupted attempts and inner review consume the same declared envelope. A new configuration or context does not renew it. Do not launch live work without caller authorization and bounds. 2. **Choose the smallest relevant measurement.** Use behavioral probes for acts, coding tasks for engineering outcomes, and separate later sessions for memory. There is no universal two-effort requirement. Keep the deployed model and effort unless the caller's decision concerns effort. Retain easy regression and cost controls; do not weaken the producer to manufacture separation. 3. **Freeze and calibrate.** Fix task, acceptance, package, model/runtime, environment and grader identities before trials. Executable oracles must accept the intended solution and reject plausible incorrect/no-op solutions. Include genuinely correct and incomplete cases when evaluating judgment. Exposed incidents are development cases, never unseen holdouts by renaming. Broken or leaked cases invalidate affected comparisons; preserve their historical disposition when versioning a correction. 4. **Run within the selected consumer's bounds.** Equalize instructions, tools and environment across arms apart from the intended variable. Coding trials expose the actual selected package and required resources. A worktree or a prompt prohibition is not runtime isolation. Exclude operator home, production tracker, session history, sibling output and solutions; capture launched configuration and final artifacts outside the worker. Report an incompatible adapter as such; do not build a replacement platform to rescue a result. 5. **Read all attempts.** Use native runner results and existing accounting; collection must not require another model call or handwritten evaluation. Keep failed, abandoned, blocked, interrupted and missing attempts visible. Wrong identity, changed acceptance, contamination or ambiguous pairing cannot establish comparison proof even when a deterministic check passed. 6. **Compare only supported facts.** Pair by task and repetition; preserve repetitions within task clusters. Report case outcomes, denominators, uncertainty and failure disposition. Endpoint reward, worker done claim, in-workflow validator PASS and independent acceptance are different facts. Missing review, usage, billing, phase or feasibility evidence stays unknown. A worker following an instruction establishes adherence, not reduced rework or causal benefit. If its task prompt repeats the skill's direction, attribute the observation to the combined instructions, not the skill alone. A passing case far from a failed boundary does not prove the boundary is repaired. 7. **Recommend once and stop.** State retain, revise, remove or insufficient evidence, the scope and supporting facts, and what remains unproven. A concrete reproduced defect with clean controls can support a provisional narrow repair; general improvement needs held-out comparison. Do not add trials until green, requir