/init

description: Set up coder-eval in this repository — scan for what is worth evaluating (Claude Code skills, an MCP server, a CLI), then scaffold a task directory with one real task and the command to run it.
disable-model-invocation: true
allowed-tools: ["Read", "Glob", "Grep", "Write", "Bash"]

Set up coder-eval in this repository

Goal: leave the user with a task directory containing **one real task** they can run immediately, not an empty scaffold. The task must exercise something this repository actually ships.

The user's request is: `$ARGUMENTS`

Step 1 — Check prerequisites

Run `coder-eval --version`. Installing this plugin did not install the CLI, and every later step needs it.

If it is missing, follow `${CLAUDE_PLUGIN_ROOT}/reference/cli-setup.md`: offer the install, **ask before running it**, and confirm with `coder-eval --version` afterwards. Never install unprompted, and do not continue if the user declines.

That reference also covers the other half of the version check — whether this project pins a coder-eval version, and what to do when the installed one does not match it.

Step 2 — Check whether this repository is already configured

This skill scaffolds a first suite. Run against a repository that already has one, it would write a "first task" beside an existing tree and report success — so find out before writing anything.

Locate the eval tree per `${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md`. **If the repository already has tasks, report the inventory and stop**: how many task files and where, how many run directories, and whether there are experiments. Then point the user at the skills that act on what exists —

• `/coder-eval:lint-tasks` to review the tasks already there;
• `/coder-eval:analyze` to read a finished run.

Only scaffold if the repository has none, or if the user explicitly asks for more after seeing the inventory. **Never overwrite an existing task file**, whatever they ask for; add alongside it.

A repository with tasks but no experiments is **partially** configured, not empty. Report that as it is and offer the missing piece — do not scaffold a first task as though there were nothing there.

Step 3 — Scan for what is testable

Look for these, in priority order, and **report what you found before writing anything**:

1. **Claude Code skills** — glob `.claude/skills/*/SKILL.md` and `**/skills/*/SKILL.md`. Skills are the highest-value thing to evaluate, because whether they trigger is invisible until it fails. If you find any, recommend `/coder-eval:check-skill` for each of them — that is a purpose-built activation suite, not something to hand-roll here. 2. **An MCP server** — an `.mcp.json`, an `mcpServers` key in `package.json` or `pyproject.toml`, or a server entry point (a `server.py` / `index.ts` that registers tools). Note which tools it exposes. 3. **A CLI entry point** — `[project.scripts]` in `pyproject.toml`, `bin` or `scripts` in `package.json`, or a `Makefile` with usable targets.

If the repository is a monorepo with many skill or package directories, cap the scan and ask which subtree to focus on rather than reporting fifty candidates.

If you find **nothing** in these three categories, say so plainly. Then offer the smallest useful thing instead — a task that runs a script or test command the repo already has and checks its output — rather than scaffolding an empty suite that proves nothing.

Step 4 — Scaffold one real task

Write into the tree step 2 resolved. If the repository has none, propose a location and ask. Step 2's never-overwrite rule still applies here — it is the one place that rule lives.

Write one task derived from what step 3 found:

• **A CLI** — a task whose prompt asks for something the CLI does, with criteria that

check the resulting file's *content*, not just that a file appeared.

• **An MCP server** — a task that exercises one specific tool and verifies its effect.
• **Skills** — point at `/coder-eval:check-skill` instead; an activation suite is a

different shape from a capability task and that skill builds it properly.

Prompts instruct, criteria validate. Do not restate in the prompt what the criteria check — a prompt that says "make sure the file contains X" tests reading comprehension, not capability.

Before writing the criteria, read `${CLAUDE_PLUGIN_ROOT}/reference/task-rubric.md`. It is the shared checklist for whether a task can pass for the wrong reason, and the one task you scaffold here is the example every later task in this repository gets modelled on — so it is worth getting right rather than fixing later. For criterion types and their fields, read `${CLAUDE_PLUGIN_ROOT}/reference/criteria.md`.

Use `/coder-eval:task` if the user wants more tasks after this one; it is the same authoring loop with a natural-language brief.

Step 5 — Environment variables

Write the variables the chosen agent needs (e.g. `ANTHROPIC_API_KEY` for the default `claude-code` agent) to **`.env.example`** — create it or append to it.

Never write `.env`: it may already hold real secrets. If `.env` does not exist, check whether it is gitignored before suggesting the user create one, and say so if it is not.

Step 6 — Validate

Run `coder-eval plan <task-directory>/*.yaml` and iterate until it exits 0. This validates the task schema through the real models, so a field name you guessed wrong surfaces here. `plan` takes task *files*: a bare directory argument is rejected outright (`Expected a YAML task file but got a directory`), so always pass explicit paths or a glob. Do not suggest `coder-eval plan` with no argument — zero-argument discovery only works from a coder-eval source checkout and exits 1 anywhere else.

An empty or task-less directory does not produce a meaningful success — if `plan` reports no tasks, treat that as a failure to scaffold, not a pass. Note that the bare-directory error is a different outcome: that one means you passed the wrong argument shape, not