/hill-climb

name: prompting:hill-climb
description: >-
  Hill-climb a skill, rule, or prompt against its eval suite. Use when changing
  one to raise an eval score, or judging whether a score change beat noise.
argument-hint: "<skill> [<suite dir>]"
allowed-tools:
  - Read
  - Write
  - Edit
  - Glob
  - Grep
  - Bash
  - Agent

Hill Climb

Goal: a skill that scores higher on its eval suite, where every accepted change carries evidence that it beat run-to-run noise and that the gain holds on cases the climb never tuned against. A change without that evidence stays out, however plausible it reads.

The suite's `README.md` names how to run it and any suite-specific rules. Follow it where it differs from this document.

Ready the Suite

Check the suite before the first baseline. A climb on an unready suite measures the harness.

• **Split.** Tag every case `dev` or `holdout` in its `case.yaml`. Candidates run on `dev`. `holdout` runs once, at the end, so no edit is tuned against it, and a climb that reads it again needs fresh `holdout` cases first. Aim for about a third of the cases in `holdout`. Give both splits a case for every kind of request the skill handles, so a `dev` guard catches a rule that bleeds from one kind into another.
• **Reach.** A case moves only when the skill fires on it. Add a `tool_used` grader on `Skill` with `input_match` naming the skill: under ablation it reports as an unscored indicator. A case where the skill never fires scores the model's default. Include cases where the skill should stay quiet, so a trigger change has a guard. The `Skill` indicator never scores, so guard a quiet case with a `trace` regex, `not_contains` on `"skill":"<name>"`. For a skill users invoke by name, simulate the invocation in `append_system_prompt` on every case, since there is no trigger to test.
• **Balance.** Include cases where the skill should change little, so a climb that over-applies the skill loses score.
• **Graders.** Grade the output with `regex` wherever a pattern decides it, and keep `llm` graders to one criterion each, with an example of a passing reply phrased differently from the obvious one, since the judge reads a bare criterion narrowly. Pair each removal grader with a survival grader for the fact that must stay, so deleting everything fails. Where a case invites invention, grade the claim of the likeliest plausible mechanism the fixture lacks, since a reply may rightly name it as ruled out. Test a claim pattern against negated forms ("haven't reproduced it").
• **Trace.** Match a tool call's input by its key and value alone. Key order in the serialized input varies by tool (`Edit` writes `replace_all` before `file_path`), so an anchored pattern can silently match nothing. Match a string value with `(?:[^"\\]|\\.)*`, since commands carry escaped quotes.
• **Scope.** When the reply wraps an artifact in commentary, have `append_system_prompt` ask for the artifact inside `<out>` tags and anchor each regex to that block, or a report that quotes a cut phrase fails its own grader. Scope a positional `llm` grader ("the first sentence") the same way, or tell the judge which part of the reply to read. `<out>(?:(?!</out>)[\s\S])*?PATTERN` finds a pattern inside the block, and `<out>(?:(?!</out>)[\s\S]){N}` holds when the block runs past N characters, a floor under `contains` and a ceiling under `not_contains`. Size N per case, since a case with more moving parts earns a longer on-topic caveat.
• **Isolation.** Stub every external service in the fixture: a bare repo for pushes, pasted text for API bodies, an `append_system_prompt` fallback telling the session what to reply when a network call fails. A run that dies on the network scores the sandbox. A case whose right answer is "no cause found" needs a fixture with no latent cause: have a strong model hunt it before the baseline. A headless session lacks interactive tools such as `EnterPlanMode` and `AskUserQuestion` even when granted. Grade a step that uses one by its observable effect, and tell the session through `append_system_prompt` that nobody answers mid-task.
• **Wrap.** An artifact outside a plugin (a user skill, an agent, a rule or `CLAUDE.md`) loads through the `wrap` key in the suite's `suite.yaml`, which the runner builds into a throwaway plugin. Context files reach the session through a `SessionStart` hook, whatever their `paths:` frontmatter says.
• **Lint.** Run `shellcheck` on the scaffold scripts, and test each regex grader with `bun evals/native/check.ts <suite>` against examples: one that should pass every grader, and one per grader that should fail it. Trace graders need `.jsonl` examples: hand-write them on the model of a smoke run's `tool_use` lines, serialized as compact JSON (`"key":"value"`, as `JSON.stringify` writes it), since trace patterns match that form. Hand-write replies for the `holdout` cases. A fix after the baseline edits a case mid-climb.
• **Noise.** Run the unchanged `dev` cases twice with an explicit `--runs` (the runner defaults to 3), priced first per `Running` at 2 × runs × the per-run cost. Compare the two with `compare.ts`. Stars there are noise at that run count, and they set how many runs a candidate needs. A star takes at least 4 runs a side at the default alpha, and `compare.ts` marks a cell below that `†`, so zero stars at fewer runs says nothing about noise. A case whose without arm swings across its range between noise columns needs more pooled runs than the rest. A grader that fails on both arms in every run is a harness artifact until a trace shows otherwise, such as a `file_exists` glob matching the sandbox's own dotfiles. Scope file globs to what a session would write. A stray `Agent` call can leave a trailing message after the reply, and a `last_message` grader grades that message, so open the trace behind a lone with-arm drop before counting it. `allowed_tools` does not remove `Agent`.
• **Headroom.** A case at 1.00 on both arms in the noise runs discriminates not