/lint-task

Lint coder-eval Task YAML

Review a coder-eval task YAML against a quality rubric and surface anti-patterns. Advisory only — does not modify files.

**Input:** `$ARGUMENTS`

• A path to one task YAML, e.g. `tests/tasks/uipath-maestro-flow/smoke/init_validate.yaml`
• Or a glob, e.g. `tests/tasks/uipath-platform/data-fabric/*.yaml`
• Or a directory, e.g. `tests/tasks/uipath-rpa/` (lints every YAML beneath it)

**Output:** One markdown report per task, printed to chat. No file writes.

This command is the source of truth for the rubric. The PR-bot workflow (`.github/workflows/lint-tasks.yml`) reads this file and applies the same rubric to changed YAMLs at PR time.

---

Severity Levels

Every issue is tagged with one of four severities:

| Level | Meaning | |---|---| | **Critical** | Test cannot meaningfully validate the skill — must fix | | **High** | Test is broken or misleading in a way that wastes infra cost or hides regressions — should fix before merge | | **Medium** | Quality gap that reduces signal but doesn't break the test — should fix | | **Low** | Polish — nice to have |

The task's overall **verdict** is the max severity across all issues raised. A task with no issues is **OK**.

Phase 1 — Resolve Targets

1. Parse `$ARGUMENTS`. 2. If it's a directory, glob `<dir>/**/*.yaml`. 3. If it's a glob, expand it. 4. If it's a single path, use it directly. 5. Skip files under `_shared/` — those are helper scripts, not task definitions. 6. If zero files match, print an error and stop.

For each resolved task file, run Phases 2–4 in parallel where possible.

Phase 2 — Read Task and Neighbors

For each target task file:

1. Read the full YAML. 2. List sibling task YAMLs in the same folder. These are the "nearby files" used for duplicate detection. 3. Read up to **5** nearby files. If more exist, pick the 5 closest by shared tags (most tag-overlap first, ties broken by alphabetical filename). 4. If the task is in a folder that only contains itself, expand the search to the parent skill directory and pick the same 5 by tag overlap.

Phase 3 — Apply the Rubric

Evaluate the task against seven axes. Each axis can produce zero or more issues, each tagged with a severity. When raising an issue, refer to the axis by its human-readable title (e.g. "Self-report anti-pattern", "Prompt over-specification") — do not use letter labels.

Self-report anti-pattern

**Raise a Critical issue if both are true:**

1. The `initial_prompt` instructs the agent to write a summary, status, audit, or report file (common names: `report.json`, `summary.json`, `result.json`, `audit.json`, `output.json`, `status.json`). Look for verbs like "save", "write", "create", "produce" near the filename. Custom names that fit the same shape ("a JSON report describing what you did", "a results file with your decisions") count too — judge semantically. 2. One or more `success_criteria` entries (`file_contains`, `file_check`, `json_check`, `file_matches_regex`) reads that same file as their evidence.

**Why it's broken:** the test should check what the agent *did* (commands run, artifacts produced) using deterministic criteria, not what the agent *claims* it did in a self-written summary. The agent grades its own homework, the result is hallucination-prone, and the deterministic criteria coder-eval was built for are bypassed.

Prompt over-specification

Evaluate how much `initial_prompt` leaks the procedure that the skill should be teaching.

Penalize:

• Step-by-step instructions in `initial_prompt` ("1. Walk the discovery hierarchy, 2. List all packages, 3. ...") — the skill should teach the procedure, the prompt should state the goal
• Prescribing specific CLI flags ("Use --output json on every uip tm command") — the skill teaches when to use flags
• Prescribing exact file paths or output formats that `success_criteria` then check (e.g. prompt says "save to `processes.json`", criterion checks `path: processes.json` exists)
• Bulleted imperatives that read like "do X, then Y, then Z" rather than "achieve goal G"

Do **not** penalize:

• Stating the high-level goal and expected output ("Use `uip` to list Flow processes and save the results")
• Naming the artifact when the agent has to know it (e.g. file paths are part of the goal, not the procedure)
• Routing context ("Use the `<skill-name>` skill workflow") — that's a skill-trigger hint, not over-specification
• **Ground-truth anchors.** Literals in the prompt that match expected values checked by `success_criteria` (e.g. prompt says "respond with 'amazing day'", criterion `file_contains` checks for `"amazing day"`). These are test fixtures — the prompt is supplying the expected output the test grades against, not leaking procedure. Same for input values the criteria assert downstream.
• **Tag-justified node/feature names.** If the prompt names a node type or feature that's also the test's `node:X` or feature tag (e.g. "Use a Switch node" on a `node:switch` task, "using a Loop node" on a `node:loop` task), that's deliberate scoping, not over-specification — the test's whole point is to exercise that node type.

Severity:

• **High** — prompt is essentially a recipe; the skill is not actually being tested because any agent could follow the recipe without invoking it
• **Medium** — prompt prescribes 2+ procedure steps or non-trivial flags
• **Low** — prompt leaks one minor detail (e.g. a single flag, a file path that wasn't necessary to specify) that is *not* covered by either carve-out above

Meaningful coverage

Evaluate whether `success_criteria` actually validate skill correctness, vs. trivial existence checks.

Penalize:

• Only `file_exists` (no content check)
• Only `command_executed` with no output validation
• `llm_judge` as the only or dominant criterion (graded by an LLM with no ground truth)
• Criteria that would pass for any non-empty / well-formed input regardless of correctness
• For `command_executed`, `min_count: 1` with a very loose regex — pr