/hill-climb

name: evals:hill-climb
description: >-
  Hill-climb a skill, agent, rule, or CLAUDE.md against its eval suite. Use
  when changing one to raise an eval score, when building or readying the suite
  a climb runs on, when judging whether a score change beat noise, or when
  testing whether an instruction still earns its tokens.
argument-hint: "<skill> [<suite dir>]"
allowed-tools:
  - Read
  - Write
  - Edit
  - Glob
  - Grep
  - Bash
  - Agent

Hill Climb

Goal: a skill that scores higher on its eval suite, where every accepted change carries evidence that it beat run-to-run noise and that the gain holds on cases the climb never tuned against. A change without that evidence stays out, however plausible it reads.

The suite's `README.md` names how to run it and any suite-specific rules. Follow it where it differs from this document. The commands for each operation below depend on the harness: [native.md](references/native.md) for `claude plugin eval`, [harnesses.md](references/harnesses.md) for any other. Use the harness the suite already runs on.

When the target is a prompt an application sends through the Claude API or the Agent SDK, run `/claude-api build-eval` to build its eval and `/claude-api hillclimb` to climb it. Those guides carry the per-model API behavior this document leaves out.

Ready the Suite

Before the first baseline, check the suite against every item in [references/suite.md](references/suite.md): the frame, case sources, the split, reach, graders, isolation, scaling, noise, and headroom. When the target has no suite yet, build one to the same checks. Re-check it whenever a case or grader changes.

Loop

Baseline

Run the suite on the base commit at least twice and pool the runs. Replicate the baseline as many times as each candidate: a single low baseline draw stars every candidate against it. Get replicates from separate invocations. Repeats inside one invocation share its drift.

Record the harness version and the model each arm ran on beside the baseline. Re-baseline when either changes mid-climb. Keep a climb log from the first run, per [references/climb-log.md](references/climb-log.md).

Error Analysis

Read the failing grader explanations, then the trace and final reply behind each. Sort each failure into one bucket:

• **Skill.** The output is wrong and the skill caused it or failed to prevent it. A candidate targets these. When the rule sits in a reference the sessions never opened, the candidate either inlines it or sharpens the pointer to it. Count the sessions that opened the reference, since injected skill text puts its filename in every trace. When another rule claims the case before the session reaches the target rule, put the fix in the text the session already follows for that case. A session that opened the rule and still hedged it also belongs here: the rule's wording lost to the model's trained default. When several such failures reproduce one outdated form, put a table mapping each old form to its replacement near the top of the body, ahead of the examples it governs.
• **Grader.** The output meets the intent and the grader failed it, or the reverse. A case that never moves while candidates fix its neighbors belongs here until its prompt is shown to state everything its graders check. Fix the grader in its own change, then re-score every stored run and compare the re-scored copies. Only a changed judge or file-target grader needs a re-baseline. Leave graders fixed during a climb, since a grader edit moves the baseline.
• **Reach.** The skill never fired, so the output is the model's default. A description-only candidate targets these. Its target is the skill-fired indicator on the cases that missed, and its guards are the quiet cases and the case scores. Fix skill failures on cases that fire first, since widening reach spreads them.
• **Floor.** The model cannot do it at this tier. Leave it.

Pick the largest skill bucket as the next target.

Candidate

Make one change per candidate, on its own branch off the base. Before running it, write down:

• The hypothesis: which instruction changes and why the traces say it will help.
• The target graders and the direction each should move.
• The guard graders a side effect would hit. A change that orders a step ("first X") needs a guard on every case where a different step must come first.

Writing the target first keeps the decision from being fitted to whichever row happened to move.

Size the change to clear the noise floor from `Ready the Suite`: rewrite the section behind a failure bucket or add the rule it lacks. A reworded line rarely moves a star.

Write the instruction as the failing behavior in general terms, never with a `dev` case's nouns, phrases, or fixture details. Write the requirement a grader checks, never the grader or the suite. Check every example the change names against the `holdout` fixtures, and swap out any a `holdout` case uses. A gain on that case would then measure the example rather than the rule.

Count the tokens the change adds to the body. A skill body re-injects at every compaction, so an added sentence needs a starred target to stay.

A removal is a candidate too, judged by whether the score holds rather than rises: follow [references/prune.md](references/prune.md). `/claude-api prompt-audit` proposes removals of text written for an older model's failures.

Screen

Screen each candidate before its full run. Run the base and the candidate concurrently on the target and guard cases only, with enough repeats in one invocation to star the smallest drop that would reject it, about 5 ÷ the drop rate a side at a ceiling base. Compare the two sides as in `Compare`. Drop or revise a candidate whose targets do not move in the predicted direction or whose guards fall, and re-screen the revision. Send the rest to verify. A screen never accepts a change. Size, invocations, and parallelism are in [references/screen.md](references/screen.md).

Ve