/benchmark-design

name: benchmark-design
description: Design and calibrate a multi-Case capability Benchmark and establish a traceable Formal Baseline.

Benchmark Design

Build a multi-Case Benchmark for one Test Agent, calibrate its difficulty with one Run per Case, and record the selected frozen Pilot as the Formal Baseline.

This Skill changes the Benchmark, never the Test Agent. It does not run or score the Test Agent. Delegate every evaluation with `run_subagent`, and tell each worker to use `agent-evaluation`. Stop after the Baseline; do not begin optimization.

Before you start

If the request does not identify a Test Agent, target capability, desired baseline score, and Pilot iteration limit, ask for the missing inputs. When they are already supplied, proceed without asking the user to restate them. Treat the current Agent as the **Builder**. A user-specified evaluation `(provider, model_id)` takes priority; otherwise inherit the current Builder Session's complete `Provider` and `Model ID` from the Environment. Never use a Project default as an implicit evaluation runtime.

Workflow

• A **Pilot** is a one-Run-per-Case evaluation used to improve the Benchmark. Unselected Pilot results never enter the Scoreboard; the selected result becomes the Formal Baseline after Freeze.
• **Freeze** means the Benchmark revision and evaluation settings stop changing.
• A **Formal Baseline** is the accepted result of the selected complete valid Pilot revision, recorded after that exact revision is frozen on one unchanged Agent State version.

Follow this order:

1. Validate the Test Agent, target capability, resolved evaluation Runtime, and evaluation access. 2. Write a Capability Contract that defines the observable process to measure, common weaker behavior, and the general Agent State improvement the Benchmark should train. 3. Plan the complete initial Case set and point allocation. For each Case, privately state the intended behavior, a plausible shortcut for a strong Test Agent, and how the Case distinguishes them. Write and leak-check the complete initial Benchmark. 4. Complete one valid evaluation for every planned Case. Together these results form Pilot iteration 1; finish this complete set before refining any Case. 5. For later Pilot iterations, use scores and Traces to reconstruct how the Test Agent solved each Case. A single iteration may refine multiple Cases or difficulty dimensions; rerun every affected Case. 6. Freeze the first valid Pilot revision that meets the desired baseline score. If none does within the requested valid-iteration limit, restore and freeze the lowest-scoring valid Pilot revision. The desired score steers refinement; the publish gate is fixed at 85 on the `0..100` scale, so a frozen revision that scores below 85 is published even when it misses the desired score. 7. Freeze the selected revision and record its complete one-Run-per-Case Pilot result as the Formal Baseline when every cell is valid, the Agent State version remains unchanged, and no known design defect remains. Do not rerun or backfill it. The Formal score does not determine validity.

Setup and access

Require a Test Agent id, target capability, desired baseline score on the fixed `0..100` scale, and a positive Pilot iteration limit. Derive a short semantic Benchmark id if needed. Resolve `(provider, model_id)` once before the first Pilot: use a user-supplied complete pair when present, otherwise inherit the current Builder Session's `Provider` and `Model ID` from the Environment. Reject a half pair or an unavailable inherited value. Read `thinking_level` from the Test Agent's `model.thinking_level` in `agent_state/system_config.yaml`, using the normal Agent-config default `medium` only when that field is absent. Do not read `thinking_level` from a Trace and do not inspect Project configuration.

The current Session must provide `run_subagent`, and the current Agent must have `agent-evaluation` installed. If either is missing, stop and explain what is needed.

Use the Environment's `App Data Dir` and the explicit Test Agent id:

TEST_AGENT_DIR = <app_data_dir>/agents/<test_agent_id>
BENCHMARK_DIR = <app_data_dir>/benchmarks/<benchmark_id>
SCOREBOARD = <benchmark_dir>/scoreboard.yaml

A Benchmark lives beside `agents/`, not inside one: it belongs to the Project and may evaluate several Agents. `test_agent_id` names the one this request evaluates, and every Evaluation records it.

Access only the specified Test Agent and Benchmark: the Agent State, complete Benchmark, and Test Traces or artifacts from valid evaluations. Do not access other Agents, Project secrets, or Evaluator State, Workspace, or Trace.

Read the Agent State version from the top-level `version` in `agent_state/system_config.yaml`; use 1 only when it is absent.

Build the Benchmark

<benchmark_id>/
├── benchmark_config.toml
├── scoreboard.yaml
└── CASE-<nnn>-<semantic-name>/
    ├── statement/
    │   ├── README.md
    │   └── <optional-public-materials>
    └── rubric/
        └── README.md

Each Case contains:

• `statement/`, which is public to the Test Agent and defines the objective, available materials, and required artifact.
• `rubric/`, which is private and defines observable scoring items, points, and Gold answers.

Both directories require a `README.md` and may contain supporting files. Do not put Gold answers for evaluated instances, hidden mappings, or private scoring conditions in `statement/`.

Create `benchmark_config.toml` with `title`, `description`, `runs = 1`, and `status = "draft"`. Benchmark design always uses one Run per Case; do not ask for or accept another Run count. `status = "draft"` tells the Web App that the Benchmark is still being built — it shows the Benchmark masked, and nobody can use or open it until the status is `published`. A draft ends in one of two states: `published` once a Formal Baseline scoring below 85 is recorded, or `failed` when calibration produces no valid