/agent-evaluation

name: agent-evaluation
description: Run one specified Test Agent on one specified Benchmark Case exactly once, privately score that execution, and return one protocol result.
version: 2026.11.03.1

Agent Evaluation

Handle one evaluation request from a `run_subagent` caller: run the specified Test Agent on one Benchmark Case once, score that execution privately, and return one protocol result.

The top-level Benchmark Designer or Optimizer owns all Case and Run loops, concurrency, and follow-up handling. This worker handles no other Case or Run, launches no evaluator or subagent, modifies no Agent or Benchmark, and never writes `scoreboard.yaml`. Use the Penguin CLI only to launch the specified Test Agent; do not use it to create another phase, designer, optimizer, or evaluator.

Operate silently. Call tools without progress messages. Across all streamed and final responses, the only worker-authored text must be the final plain protocol YAML. Emit no narration, headings, Markdown fences, summaries, private scoring details, or other text.

For the caller

The agent that fans out the cells — a Benchmark Designer, an Optimizer, or an agent asked to evaluate a Test Agent — fixes `provider` and `model_id` once, before its first `run_subagent`, and sends that pair in every request of the evaluation. It is the pair the caller's own instructions name (the user's request, or an Optimizer's Reference evaluation); when they name none, it is the caller's own Session's model, the `Provider` and `Model ID` lines of its Environment. An Agent stores no model — the `model` block of its `system_config.yaml` holds limits and a thinking level, never a provider or model id — so never look for one in the Test Agent's files, never fall back to a Project default, and never ask the server for one. When neither source gives a complete pair, stop and ask the user. The thinking level is not the caller's to choose: every worker reads it from the Test Agent's configuration. A Benchmark whose Case statements say the case is run with Harbor from a benchmark repository (a `## How this case is run` section naming Harbor, a repository link and the launch) adds the caller's steps in `reference/harbor.md` §A.

A caller never reads a credential — the server's `api-token` file, a Project's `.project_config.toml` (it holds the model keys), the server's auth database `web.db`, or a vault — and never calls the server's HTTP API with a token read from disk. Workers are bound by the same rule (Contract).

Before you start

Use this Skill only for a complete request from a `run_subagent` caller. If the request is incomplete or inconsistent, return `invalid_request` through the protocol instead of asking the user a question.

Contract

Require exactly one value for every field below:

protocol_version: 1
case_id: <case_id>
run: <1_based_run_index>
expected_version: <tested_agent_state_version>
test_agent_id: <test_agent_id>
benchmark_id: <benchmark_id>
provider: <provider>
model_id: <model_id>

One request represents one Test Agent execution. The `run` value identifies that execution; it is not a repeat count. `provider` and `model_id` must both be non-empty and select that exact configured model. If a required field is missing, duplicated, or conflicting, return `invalid_request` without creating a Workspace or launching the Test Agent.

Return a **scored result** when the Test Agent ran and the Rubric could be applied. Wrong, malformed, or missing Test Agent output is still a scored result. Return an **evaluation failure** when the request, Benchmark, launch, version check, Trace binding, or scoring process prevents a valid score.

Resolve the Project, Test Agent, Benchmark, and Case only from the explicit request and Environment App Data Dir. Reject traversal, symlink escape, or any path outside the requested Test Agent and Benchmark. Never read a Project configuration file, credential, or vault: not the server's `api-token` file, not `.project_config.toml` (it holds the model keys), not the server's auth database `web.db`. Never call the server's HTTP API with a token read from disk either. The one exception is the model API host of a Case run through Harbor, which the benchmark repository's helper prints for you (`reference/harbor.md` §B.3); no other part of the configuration reaches you. Never use `penguin config model list` for it either: that listing masks each key but still shows its last 4 characters.

Prepare

Use the `App Data Dir` from the Environment:

TEST_AGENT_DIR = <app_data_dir>/agents/<test_agent_id>
BENCHMARK_DIR = <app_data_dir>/benchmarks/<benchmark_id>

The Benchmark is Project-level and is not owned by the Test Agent: it sits beside `agents/` and may evaluate several Agents. `test_agent_id` names the Agent this request evaluates; return it as `agent_id`.

Reject path traversal, symlink escape, or any resolved path outside the requested Test Agent and Benchmark. Inspect only the requested Agent State, Benchmark config and Case, isolated Test Workspace, and Traces needed to verify this execution. Do not inspect another Agent, Project secrets, hidden configuration, or unrelated Workspaces or Traces.

Require `agent_state/system_config.yaml`, `benchmark_config.toml`, `<case_id>/statement/README.md`, and `<case_id>/rubric/README.md`. Return `benchmark_invalid` when `benchmark_config.toml` says `status = "failed"`: a Benchmark whose calibration failed is not evaluated. Treat `run` only as the caller-owned label for this evaluation and return it unchanged; do not read or validate the total Run count. The top-level Agent State `version`, defaulting to 1, must equal `expected_version`; otherwise return `version_changed`. Read and snapshot `model.thinking_level` from this Target Agent config, using the normal Agent-config default `medium` only when the field is absent. This configured value is the evaluation `thinking_level`; do not require or read thinking metadata from a