/coder-eval-create-plan

description: Create a structured, phased implementation plan for a feature or change in the coder_eval codebase, executable from a fresh session by /coder-eval-implement-plan

Context

• Current git status: !`git status --short`
• Current branch: !`git branch --show-current`
• Recent commits: !`git log --oneline -5`

Your task

Produce a detailed, phased plan that a **fresh session with no memory of this conversation** can execute via `/coder-eval-implement-plan`. The plan is the contract between you and the implementer — it must be self-contained. The implementer drives directly off these parts, so each must be concrete:

• **Per phase**: `Changes`, `Edge Cases`, `Tests to Write`, `Tests to Run` (real scoped `uv run pytest …` commands), `Acceptance Criteria` (verifiable, with checkboxes), and a **`Risk` tag** (Low / High — sets the implementer's review depth).
• **Global**: `Patterns to Mirror` (actual code snippets from this repo, not descriptions), `Design Context`, `Master Acceptance Checklist`, `Confidence Score`.

The input may be a feature description, a file path to a spec/design doc (read it first), a bug list, or a combination.

Follow these steps:

1. **Gather input** — If the user references a file, read it in full first. If it's a bug list, enumerate each item. If it's a direct description, use it as-is.

2. **Understand & classify** — Restate the goal in one or two sentences. If there are multiple items, summarize scope and list each. Then classify:

| Level | Indicators | |-------|-----------| | Small | 1-3 files, follows existing patterns, <100 new lines, single phase | | Medium | 3-10 files, one new criterion / one CLI flag group / one model addition, 2-4 phases | | Large | 10+ files, new agent or cross-module interaction (orchestrator + models + criteria), 5+ phases | | XL | Architectural change, new subsystem, migration of an existing contract — split into multiple plans |

For **Small**, collapse to one phase (or a flat task list). For **XL**, stop now — present the classification rationale and a proposed split, and wait for user confirmation before any further research or planning.

3. **Research the codebase** — Read all relevant files to understand the current state. Implement by mirroring and re-using existing patterns, not inventing. Pay special attention to:

• `coder_eval/models/` — Pydantic data models (all importable from `coder_eval.models`; declared once, consumed everywhere — SSOT)
• `coder_eval/criteria/` — Plugin registry with auto-discovery via `@register_criterion`; `SuccessCriterion` discriminated union in `models/criteria.py`
• `coder_eval/agents/` + `coder_eval/plugins.py` — Agent ABC implementations registered through the **`coder_eval.plugins` entry-point SPI** against `AgentRegistry`. `agent.type` is an open string; `Orchestrator._create_agent` delegates to the registry's `create_agent()` factory (`agents/registry.py`) instead of dispatching by kind, so you add an agent via a `register(registry)` hook — **not** by editing `_create_agent` or the `AgentKind` enum (`models/enums.py`, which lists only the known built-in kinds)
• `coder_eval/orchestration/` — Batch execution, experiment resolution, and the single declarative merge resolver (`config_merge.py::resolve_root`)
• `coder_eval/cli/` — Typer + Rich CLI commands; generic `-D`/`--set` overrides (`orchestration/overrides.py`)
• `coder_eval/streaming/` — Real-time agent event streaming; the agent is the sole emitter and `EventCollector` is the single TurnRecord capture seam
• `tests/lint/rules/` — custom architectural lint rules (CE001–CExxx), the project's harness for mechanically-enforced invariants

Compile findings into **Patterns to Mirror** with actual snippets and `file:line` references, each tagged `APPLIES_TO: Phase N`.

4. **Think through the design** — Before writing phases, reason explicitly about:

• Does this change touch the evaluation flow? (CLI → ExperimentRunner → run_batch → Orchestrator → Sandbox + Agent + SuccessChecker)
• Does this affect the 5-layer config merge? (default.yaml → experiment defaults → task YAML → variant → CLI flags). Each list/dict field must declare its `MergeField` strategy (lint rule CE014).
• If adding a new criterion: does it fit `BaseCriterion` / `@register_criterion` / the `SuccessCriterion` discriminated union? Does it need a custom `aggregate()` for suite thresholds?
• If adding a new agent: does it follow the plugin SPI (a `BaseAgentConfig` subclass + `Agent` ABC + a `register(registry)` hook exposed via the `coder_eval.plugins` entry-point group)? Does it use the shared turn lifecycle (`_begin_turn`/`_end_turn_ok`/`_mark_stopped`) and emit the standardized event protocol?
• Does this change the task YAML schema? If so, what happens to existing task files in `tasks/`?
• Are there edge cases in sandbox isolation, agent lifecycle, retry/crash recovery, or token accounting?
• Does this introduce new dependencies? Prefer what's already in the project (pydantic, typer, rich, anyio, anthropic).
• Could this break existing experiments or evaluation results?

5. **Simplicity & reuse gate (KISS / DRY / SSOT / YAGNI)** — Before writing phases, audit the intended design against the project's Design Principles (in `CLAUDE.md`) and resolve tensions *with the user*, not silently:

• **KISS / YAGNI** — Is any phase introducing an abstraction, config layer, or generality the *current* requirement doesn't need? Is there a simpler shape (a constant instead of a config field, a direct call instead of a new layer, mirroring an existing criterion instead of a new base class)?
• **DRY** — Does any new field type, constant, validation rule, helper, or model duplicate something that already exists? (You scanned for helpers during research — now decide reuse-vs-extract for each.)
• **SSOT** — Is each piece of knowledge (enum values, field constraints, criterion discriminators, pricing