/skill-eval

name: skill-eval
description: 'Measure whether a skill helps by comparing runs with and without it. Use when: reading skill A/B results or deciding to keep, revise or remove one.'
practices:
- measurement-over-assertion
- ab-testing
skill_api_version: 1
hexagonal_role: supporting
consumes:
- skill-source-package
produces:
- skill-maintenance-recommendation
- probe-package
context_rel:
- kind: supplier-to
  with: skill-builder
user-invocable: true
metadata:
  tier: meta
  dependencies: []
  capabilities: ["evaluate_skill_decision","select_skill_measurement","read_skill_ab_results","author_seeded_probe"]
  effects: ["write_probe_package","dispatch_authorized_trials"]
  canonical_status: canonical
  disposition: keep_specialist
  stability: experimental
output_contract: one recommendation (retain, revise, remove or insufficient evidence) with cases, all-attempt denominators, paired outcomes, uncertainty, cost and unknowns

Skill Eval

Answer one named maintenance decision: **retain, revise, remove, or insufficient evidence**. Choose the measurement that can answer that decision, use the caller's accepted cases and resource envelope, make one scoped recommendation, and stop. A completed evaluation does not require a positive difference.

This is an optional specialist: the selected runner owns execution and bounds, and its native results own the measurements.

Rules that decide the answer

• **Count every attempt.** Keep failed, crashed, interrupted, blocked, abandoned,

missing and infrastructure-invalid attempts in the all-attempt accounting. A rerun adds an attempt; it never overwrites the one that failed. A run that shared its host with other work still counts for completion and tokens; keep its duration, labeled as measured on a shared host, rather than dropping it. Its timeout is not by itself a skill failure; when timeouts fall unevenly between arms, report them as a possible skill effect, not infrastructure.

• **Vary one thing.** Equalize instructions, tools, environment, model and

effort across arms apart from the intended variable. If one arm's task prompt repeats the skill's direction, attribute the result to the combined instructions, not the skill alone. When you control launch order, alternate arms run by run, swap which goes first in each pair and rerun a dead run in its own slot; one whole arm and then the other turns load, rate limits and time of day into an apparent skill effect.

• **Shape the case as the caller does.** For a skill another skill or a

standing instruction invokes, word the task as that caller passes it, put pre-existing items of the same kind in the files the run touches, and have the grader fail work outside the asked scope: unrequested edits, new files, commits. A looser task credits the skill with the agent's own extras, and a clean tree lets "touch only the current work" pass trivially.

• **Locate the change against what the harness fixes.** A replayed transcript,

pinned stub or frozen upstream output hides any change made before that point; report it as not measured. A defect that appears only after compaction or resume needs a case holding only what survives; a full-context run can pass both arms.

• **Confirm the skill loaded.** Before reading a zero or small delta as no

benefit, check each treatment run for the skill actually being loaded or injected. A run where it never loaded measures routing, not content. When the act depends on a rule kept in a reference, also check that the run opened that file; a run that did not still counts; read its miss as the body's pointer to the reference failing, not the reference's rule, and report how many treatment runs opened the file. The behavioral probe injects only SKILL.md or a prelude, so it cannot answer this.

• **Screen each case for headroom first.** A case the no-skill arm already

passes nearly in full cannot show a benefit: report it as no headroom, not no value, keep it only as a regression or cost control, and write a harder case ([seeding](references/seeding.md)). Read a perfect score in every run of either arm as a suspect test until the grader check below shows it can fail.

• **Calibrate the grader.** Before trusting scores, confirm the judge or

discriminator passes a response that plainly meets each criterion and fails one that plainly does not. A weak judge can fail correct responses wholesale.

• **Name the model and harness.** A result holds for the model, effort,

runner and host it ran on. The same model can score several points apart on another harness or machine, so do not carry a result to a different model or runner, and do not compare arms run on different ones.

• **Small samples are directional.** Report uncertainty with every difference.

A difference without it shows neither benefit nor equivalence, and a zero-crossing interval is not equivalence. For two unpaired success rates, give each arm a Wilson interval, the difference with a Newcombe hybrid score interval and Fisher's exact test, not a normal approximation, which breaks at 0/n, n/n and a handful of runs. AgentOps ships no tool for this; the stock calls need the third-party scipy and statsmodels Python packages: `scipy.stats.binomtest(k, n).proportion_ci(method="wilson")`, `statsmodels.stats.proportion.confint_proportions_2indep(k1, n1, k2, n2, method="newcomb")` and `scipy.stats.fisher_exact`. If they are absent, say so; do not fall back to a normal approximation. Beside an inconclusive result, state the runs per arm needed to detect the smallest effect that would change the decision, and return insufficient evidence; running them is a new measurement with its own fixed stop and the caller's authorization, never an extension of this one.

• **Fix the stop before running.** Do not add trials until the result turns

positive, remove losing observations or relax acceptance.

• **Know the noise before cutting.** Before trimmin