/generate-task

Generate Test Task

Generate **one** coder-eval task YAML (and an optional check script) for the scenario described in `$ARGUMENTS`.

**Input:** `$ARGUMENTS` — a free-form description of the scenario the task should cover. The target skill is **always inferred** from the description (Phase 1a). Do not require or accept a skill name as a separate argument.

**Optional tier filter.** If the description contains a bare `smoke` / `integration` / `e2e` token (whitespace-separated), use it as the tier hint. Otherwise pick the tier from the scenario itself per the Phase 2a table.

**Output:** ONE task YAML at `tests/tasks/<skill-name>/...` plus an optional `check_*.py` for e2e tasks. Always one task per invocation — re-run the command for additional scenarios.

> ## ⚠ Generated tasks are unverified scaffolds > > The author **MUST** before merging: > 1. Run the task end-to-end with `coder-eval` (command in the Phase 4 summary). > 2. Confirm it passes. > 3. State that explicitly in the PR description (e.g. `Ran <task-id> locally and it passed.`). > > The lint workflow at `.github/workflows/lint-tasks.yml` raises a **High** issue when the PR description doesn't claim a passing run. Pushing the YAML alone is not enough.

---

Phase 1 — Context Gathering

1a. Infer the target skill from the description

1. List candidates via `ls skills/uipath-*/SKILL.md`. 2. Read every `SKILL.md` frontmatter (name + description) — that is what the runtime uses to decide which skill activates on a given prompt. 3. Match the input description to the skill whose frontmatter best covers it. Tiebreak by inspecting the candidates' `references/` and `assets/` filenames. 4. If two or more skills are plausible, ask the user to disambiguate before proceeding. Do not guess.

Once resolved, `<skill-name>` is fixed for the rest of the run.

1b. Read context (parallel Explore agents or parallel tool calls)

1. `skills/<skill-name>/SKILL.md` plus everything under `skills/<skill-name>/references/` and `skills/<skill-name>/assets/`. 2. `tests/README.md` — authoritative source for the **Tag Taxonomy**, **Weight scale**, and **experiment defaults** (`smoke.yaml` for PR-gate, `default.yaml` for nightly e2e and ad-hoc, `smoke-windows.yaml` for Windows RPA). Do not duplicate that material here; reference it. 3. `.claude/commands/lint-task.md` — the quality rubric (six axes, four severities). Generated tasks must not trip a Medium-or-above issue on any axis. 4. `tests/reports/<skill-name>.md` if it exists — use it to detect that the scenario in the description is already a known gap with prior recommendations. 5. Every existing `*.yaml` task under `tests/tasks/` — collect all `task_id` values (collision check), study conventions for the target skill (or a peer skill if the target has none yet), and confirm the described scenario is not already covered.

Phase 2 — Task Design

The described scenario IS the task. If an existing task already covers it, stop and tell the user — do not generate a duplicate. Suggest the existing task; if the existing task is weak (Medium-or-above issues against `lint-task.md`), suggest amending it instead of adding a new one.

2a. Tier

Per the tier table in `tests/README.md`:

| Signal | Tier | |---|---| | CLI produces correct output, report generation | smoke | | Multi-step workflow with error handling, cross-component integration | integration | | Full build → validate → run cycle, artifact correctness | e2e |

2b. Tags

Required, in this order: `<skill-name>` first, then `<tier>`, then `mode:<X>`. Optional dimensions (`shape:*`, `node:*`, `resource`, `connector`, `windows`, `feature:*`, `path-to-ga`) follow per `tests/README.md#tag-taxonomy`.

• **`<skill-name>`** — always the first tag. Must match the `skills/<name>/` folder exactly (e.g. `uipath-maestro-flow`, `uipath-agents`). Never abbreviate or drop the `uipath-` prefix.
• **`<tier>`** — `smoke`, `integration`, or `e2e`.
• **`mode:<X>`** — required per the Coding Agents Scorecard tagging system (introduced in skills PR #614 by Tomasz Religa). Pick exactly one:
• `mode:build` — creating, designing, editing, or deploying flows, nodes, or skills.
• `mode:operate` — running, triggering, managing live instances, connectors, or integrations in production.
• `mode:diagnose` — investigating faults, inspecting execution traces/variables, evaluating, or debugging.

**Use only the closed-vocabulary values listed in `tests/README.md`.** If no value fits an optional dimension, omit it and surface it in the Phase 4 summary so the taxonomy can be extended deliberately. Never invent tag values inline.

Final tag order: `[<skill-name>, <tier>, mode:X, shape:X, node:..., resource, connector, windows, feature:..., path-to-ga]`.

Example:

tags: [uipath-maestro-flow, e2e, mode:build, shape:multi-node, node:decision, connector, feature:http]

2c. task_id

Convention: `skill-<domain>-<capability>` (lowercase kebab-case, see `tests/README.md` for the domain mapping). Verify uniqueness against the IDs collected in Phase 1.

2d. initial_prompt

State the **goal and expected output**. The skill teaches the procedure — that is what the test is verifying. Mirror the prompt style of existing tasks for the target skill.

Two anti-patterns from `lint-task.md` to actively avoid:

• **Self-report anti-pattern (Critical).** Never instruct the agent to write a summary/status/report file that the success criteria then read for evidence. Criteria must check what the agent *did* (commands, real artifacts), not what it *claims* it did.
• **Prompt over-specification (High/Medium/Low).** No step-by-step procedure, no prescribing flags or exact output formats unless they are ground-truth anchors that the criteria assert. Naming a node type that matches a `node:X` tag is fine — that is deliberate scoping.

Keep prompts concise (typically under 3 lines).

2e. success_criteria

Pick from the criteria types documented in `tests/README.md` and