/task

description: Turn a natural-language description into coder-eval task YAML — minimal prompts, weighted criteria that check output content, validated with `coder-eval plan`. Use when the user wants to write, add, or generate an evaluation task.
allowed-tools: ["Read", "Glob", "Grep", "Write", "Bash"]

Author a coder-eval task

You are writing coder-eval task YAML. The user's request is: `$ARGUMENTS`

If `$ARGUMENTS` is empty, ask what the task should test. Do not invent a subject.

Good tasks use simple prompts: state the goal and the expected output, then let the agent work out the approach. A single request can produce **several** task files — "create tasks for all the registry subcommands" means one task per subcommand.

Step 1 — Understand the request, and check the CLI is there

Run `coder-eval --version` first. Steps 6 and 7 both shell out to it, and finding that out *after* writing several task files means the user gets a bare `command not found` with nothing to act on. Installing this plugin did not install the CLI.

If it is missing, follow `${CLAUDE_PLUGIN_ROOT}/reference/cli-setup.md`: offer the install, **ask before running it**, and confirm with `coder-eval --version` afterwards. Never install unprompted, and do not write any task files if the user declines.

That reference also covers the other half of the version check — whether this project pins a coder-eval version, and what to do when the installed one does not match it.

Then establish:

• **What is being tested** — which tool, SDK, CLI, skill, or capability?
• **How many tasks** — one operation, or several?
• **Difficulty** — smoke, basic, or intermediate?
• **Dependencies** — network, packages, starter files, external services?

State any assumptions you make rather than silently picking.

Step 2 — Look at what already exists

Find the repository's task tree by following `${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md`, and say what you resolved. If a task already covers this ground, say so and offer to modify it instead of adding a near-duplicate.

**Repo-local convention beats anything bundled with this plugin — where the two disagree, the repo wins.** Before writing, read what the repository declares about task authoring: its own contributor or convention documents, a task template if it ships one, and a few neighbouring tasks. Adopt what you find — naming, tags, thresholds, weights, where files go — and **say in your report which conventions you adopted**, so the choice is visible rather than implied.

Two limits on that:

• **A repository that declares nothing** leaves the bundled rubric as the whole answer.

Precedence is about deferring to a local rule that exists, not about doing nothing until one does.

• **Precedence covers style, not soundness.** If a local convention would produce a

criterion that cannot fail, the rubric's correctness checks still bite — follow the convention where you can, say plainly where you did not and why.

Step 3 — Design the task

**Task ID** — lowercase kebab-case, unique, `<domain>-<action>` (e.g. `registry-list-processes`).

**Initial prompt** — minimal. State the goal and the expected output; nothing else.

• Good: "Use the `foo` CLI to list the available processes and save the result to

`processes.json`."

• Bad: a step-by-step recipe with the exact flags, or a restatement of what the

criteria check.

**Key rule: prompts instruct, criteria validate.** Never leak criteria detail into the prompt. If a criterion checks that the output contains a `count` field, the prompt must not mention `count` — otherwise you are testing transcription, not capability.

The subtle version of this, and the easiest to write by accident: a criterion that matches a literal the prompt already dictates. "Use `pypdf` to read the fields" in the prompt plus a criterion grepping for `pypdf` is a criterion that cannot fail — the agent was told the answer. Either the constraint is a real requirement (keep it in the prompt, and score what the agent *did with it* instead) or it is the thing under test (drop it from the prompt). Never both.

(The rubric below carries this same trap as a review-time check, and is the declaration a reviewer applies. The paragraphs above are the authoring-time version: they exist to stop you writing it in the first place.)

**Success criteria** — read `${CLAUDE_PLUGIN_ROOT}/reference/task-rubric.md` *before* choosing them. It is what this work will be checked against in step 5, and a criterion set designed against it is far cheaper than one repaired after the fact.

Pick by what actually needs verifying:

| What to check | Criterion type | | --- | --- | | File exists, has content, matches a pattern | `file_check` (prefer over `file_exists` + `file_contains`) | | JSON structure or specific values | `json_check` (JSON Schema + JMESPath assertions) | | A script runs, tests pass, or a scorer emits a float | `run_command` | | Output resembles a reference solution | `reference_comparison` | | Subjective or open-ended quality | `llm_judge` | | A deep, tool-using verdict on the sandbox | `agent_judge` (expensive) | | The agent used a specific tool | `command_executed` | | Tool-call efficiency against a budget | `commands_efficiency` | | The agent engaged a target skill | `skill_triggered` (see `/coder-eval:check-skill`) | | A predicted label vs. ground truth | `classification_match` |

Read `${CLAUDE_PLUGIN_ROOT}/reference/criteria.md` for each type's exact fields — it is generated from coder-eval's own models, so it is the authoritative field list.

Rules that matter:

• **Every task needs at least one criterion that checks output *content***, not just

existence. A suite of `file_exists` checks passes when the agent writes an empty file.

• Use `command_executed` sparingly — only when it genuinely matters *how* the result was

produced. Set `require_success: true` whenever the command's success is what you are grading; the permissive default (`false`) cou