/coder-eval-run-analysis

allowed-tools: Read(*), Glob(*), Grep(*), Bash(ls:*), Bash(wc:*), Bash(jq:*), Bash(python3:*), Write(runs/*)
description: Analyze a coder-eval run and write analysis.md with systemic patterns, findings, and concrete fixes

Context

You analyze a coder-eval run and write `analysis.md` to the target directory. The target path is `$ARGUMENTS` (default `runs/latest` — inform the user if you fell back to the default).

**Do all reasoning yourself in this session — no sub-agents, no `Agent` tool calls.** Read files in parallel (batch Read calls in a single assistant turn) and write the report inline.

Run layout and scope-marker files (`run.json` / `variant.json` / `experiment.json`) are defined in `.claude/shared/run-layout.md` — read it; Step 1 below relies on those markers.

Step 1: Determine scope

Inspect the target path:

• `task.json` directly inside → **task scope** (single replicate).
• `??/task.json` subdirs but no `variant.json` → **task scope** (aggregate replicates per `task_id`).
• Contains `variant.json` → **variant scope**.
• Contains `run.json` → **run scope**. If `experiment.json` also exists, this is a multi-variant experiment.

Step 2: Read data

**Task scope (single)**: read `task.json`.

**Task scope (aggregate replicates)**: read every `??/task.json` and merge — per-replicate arrays for `final_status`, `weighted_score`, `iteration_count`, `duration_seconds`, `total_cost_usd`; union of `success_criteria_results` keyed by criterion `description`. Drive recommendations from the aggregate ("3/5 replicates failed criterion X"), not cherry-picked replicates.

**Variant / run scope, > 20 tasks**: do NOT read the full `task.json` files — the `turns` arrays are large and only useful per-task. Use `jq` (or `python3` if missing) to extract a compact summary per task:

{task_id, final_status, weighted_score, duration_seconds, total_cost_usd,
 total_tokens, assistant_turn_count, max_turns, max_turns_exhausted,
 iteration_count, model_used, criteria_count, all_criteria_perfect,
 failed_criteria: [{type, description, score, error_excerpt}]}

`error_excerpt` = first ~200 chars of each failing criterion's `error` / `output` / `Instructions` field. This is what enables clustering in Step 3.

**Variant / run scope, ≤ 20 tasks**: read all `??/task.json` files directly.

Also read `run.json` (run scope), `variant.json` (variant scope), and `experiment.json` + `experiment.md` (experiment runs).

Step 3: Analyze

Apply these seven dimensions to your reasoning — diagnose lens for failures, optimize lens for passes:

1. **Outcome** — failed: root cause (`prompt_gap` / `environment_issue` / `agent_error` / `config_issue` / `impossible_task`), which criteria failed and by how much. Passed: are all criteria at 1.0 (task too easy)? 2. **Prompt** — failed: missing context / flags / paths / identifiers, mismatch with what criteria check. Passed: over-specified, hand-holding, unnecessary verbosity. 3. **Agent efficiency** *(task scope only)* — turn utilization, command patterns, error recovery, stuck-in-loop, slow commands, file-output timing. Skip at variant/run scope. 4. **Criteria** — sensitivity `weight × (threshold − score)`; fragile passes at threshold; redundant / coverage-gap criteria. 5. **Configuration** — lineage conflicts (`source != "task"`), `max_turns` hit / excessive, model fit, `allowed_tools` alignment. 6. **Environment** — infra errors, missing services, expired credentials, CLI tool errors. 7. **Cost & performance** — token breakdown, cache hit rate (`cache_read / (cache_creation + cache_read)`), cost reasonableness, cost-per-score-point, duration headroom.

Task scope flow

Apply all seven dimensions inline to the single (possibly aggregated) task and produce the task-scope report.

Variant / run scope flow (pattern-first)

For > 20 tasks, **cluster before deep-diving**. This is the main work saver — most run-scope failures share a small number of root causes.

1. **Cluster failures** by `(failing_criterion_signature, error_excerpt_fingerprint, score_signature)`. A cluster of ≥ 3 tasks becomes a **Systemic Pattern**: root-cause hypothesis, affected task list, representative evidence quote, recommended fix (CLI / env / criteria / prompt), estimated score recovery. Track the union of covered task_ids as `pattern_task_ids`. 2. **Individual findings** for failed tasks NOT in `pattern_task_ids`, capped at the top **K = 15** by `total_cost_usd`. Apply dimensions 1, 2, 4 (task design / prompt / criteria). Singletons below K are implicitly covered by Cross-Task Common Findings. 3. **Aggregate sections**:

• **Efficiency Ranking** — top 10 failed tasks by `total_cost_usd` with score / turns / duration / cost / one-line note.
• **False Negatives** — tasks where output-file evidence shows success but `command_executed` / similar criteria reject (e.g., alternative-but-valid commands, case mismatches).
• **Cross-Task Common Findings** — patterns across tasks below the ≥ 3 systemic-pattern threshold.

For ≤ 20 tasks, skip clustering and apply all seven dimensions inline per task.

Run scope with experiment.json (multi-variant)

Additionally produce — pull aggregates from `experiment.json`, do not recompute p-values, win rates, or score spreads:

1. **Experiment Summary Table** — scores, durations, p-values from `experiment.json`; add cost totals per variant from `run.json.task_results`. 2. **Efficiency Comparison** — per-task score + cost + duration per variant; cost-per-score-point. 3. **Variant Recommendation** — one paragraph: "Pick `<variant>` because…" with score / cost / speed tradeoff. 4. **Task Difficulty Ranking** — rank by `score_spread` from `experiment.json.task_summaries`. Zero spread = not discriminating; high spread = good discriminator. 5. **Failure Clusters** — group failed tasks across variants by root cause.

Step 4: Synthesize and write

Before writing, apply:

1. **Already-fixed check** — for each YAML recommendation, read th