evals

Eval suites

**Navigation**: [Agents package rules](../AGENTS.md) → **Eval suites**

Paths are relative to `packages/agents/` unless they name another package.

Graph authoring eval suite (`graph-planner`)

`src/evals/` carries a provider-agnostic evaluation harness for graph authoring: `GRAPH_PLANNER_EVAL_CASES` (objectives + structural expectations — input wiring, node-family patterns, branch handles, reachability, prompt text, no provider-locked nodes) and `runGraphPlannerEval`, which drives `authorGraph` (metrics per case: accepted, score, authoring rounds, tool calls, duration, cost; aggregate: success rate, one-shot rate, averages). Run it against any registered provider:

Two metrics were mapped when the suite moved off the retired one-shot planner: **authoring rounds** replaces submit rounds and counts `execute_code` actions, so a one-shot is a graph delivered in the first action; **attempts** is gone, because `authorGraph` has no outer retry loop — a repair is another authoring round.

npm run dev:nodetool -- eval graph-planner --list
npm run dev:nodetool -- eval graph-planner -p anthropic -m claude-sonnet-5
npm run dev:nodetool -- eval graph-planner -p ollama -m qwen-3.5:4b --cases summarize
npm run dev:nodetool -- eval graph-planner -p openai -m gpt-5.4-mini --json --out report.json
npm run dev:nodetool -- eval graph-planner -p anthropic -m ... --min-success 0.8  # floor

**Where the suites run.** On request. `.github/workflows/agent-eval.yml` (`workflow_dispatch` only) runs the suites that need a provider, plus `jtbd`, against inputs for suites/provider/model/cases and a floor override, uploads each suite's JSON, and renders one table. Nothing here gates a merge: a model run costs money and carries run-to-run variance, so it is a tool a maintainer starts after changing the planner, the CodeAct loop, the sub-agent path or a `ui_*` contract. The free half is a local command — `nodetool eval <suite> --keyless` runs the cases that need no key and no network, and `--list` marks them.

Harness tests (scripted provider, no network): `tests/graph-planner-eval.test.ts`.

End-to-end eval suite (`graph-e2e`)

The `graph-planner` suite stops at the graph: it scores structure, which says nothing about whether the workflow does what was asked. `src/evals/graph-e2e- {cases,eval}.ts` closes that loop — every case runs three phases and only counts as a success when all three hold:

1. **plan** — `authorGraph()` produces a graph, scored structurally by the same `checkExpectations` the graph-planner suite uses. 2. **execute** — `applyRunPolicy` (`src/run-policy.ts`) stamps the run's provider/model onto the planner's Agent nodes (the planner leaves them model-less on purpose — the run owns that choice — so an unstamped graph dies on "Select a model"). This suite is its only importer, and the reason the function outlived the retired `Agent` graph branch, then the graph runs for real with the case's inputs as run params, through a caller-supplied `GraphRunner`. The runner is injected so this package needs no execution dependency and the harness tests can drive scripted runs with no kernel; the CLI wires the real one over `ExecutionSession` (`packages/cli-harness/src/evals/graph-runner.ts`). 3. **judge** — deterministic output checks (an output by name exists, is non-empty, matches/doesn't match a literal) plus an LLM judge (`src/evals/goal-judge.ts`) that reads the case's goal statement and the actual outputs and answers `{achieved, score, reasoning}` as plain JSON. A regex cannot tell a real German translation from the English echoed back; the judge can. A provider failure or unparseable answer is reported as a judge error, never as a pass.

Metrics per case: planned, executed, goalAchieved, score, submit rounds, node/ edge counts, plan and run duration, cost, plus the outputs themselves. Aggregate: end-to-end success rate (the `--min-success` gate), plan rate, execution rate, mean score. Cases whose graph needs a real model (`needsModelProviders`) skip without configured providers; the two deterministic cases (`concat`, `arithmetic`) run anywhere and use `skipJudge`, since their outputs are pinned exactly by pattern.

Each case costs inference twice — the run, then the judge — so it is the most expensive suite here. A full pass on `claude_agent_sdk`/`sonnet` runs ~$0.07.

npm run dev:nodetool -- eval graph-e2e --list
npm run dev:nodetool -- eval graph-e2e -p anthropic -m claude-sonnet-5
npm run dev:nodetool -- eval graph-e2e -p openai -m gpt-5.4-mini --cases concat,arithmetic
npm run dev:nodetool -- eval graph-e2e -p anthropic -m ... --timeout 600000 --min-success 0.8

Harness tests (scripted provider, fake runner, no network): `tests/graph-e2e-eval.test.ts`.

Code node authoring eval (`code-gen`)

`src/evals/code-gen-{cases,eval}.ts` drives the real `CodePlanner` over eight cases — one per authoring shape the feature targets (reshape, merge/join, compute, extract/parse, split, format, validate, seed) — and scores each accepted `submit_code` submission structurally: declared outputs present and typed, inputs limited to the slots the dialog offered, the destination-handle case's expected output present with the right type, every declared output assigned on every visible return path (`analyzeGeneratedCode`), no `state`/ `yield` when nobody asked, and no name that is neither sandbox API (`unknownApiReferences`) nor bound by the code itself (`collectBoundNames`).

Acceptance is reported twice: **first-pass** (accepted on round 1, before the tool fed anything back) and **post-repair** (accepted at all within the round cap). `--min-success` gates on post-repair.

npm run dev:nodetool -- eval code-gen --list
npm run dev:nodetool -- eval code-gen -p anthropic -m claude-sonnet-5
npm run dev:nodetool -- eval code-gen -p openai -m gpt-5.4-mini --min-success 0.9

Harness tests (scripted provider, no network): `tes