/ci

description: Generate a GitHub Actions workflow that runs a coder-eval suite as a CI gate or on a schedule, using the published composite action — with the agent runtime, credentials, JUnit output and the run reports wired correctly.
disable-model-invocation: true
allowed-tools: ["Read", "Glob", "Grep", "Write", "Bash"]

Wire coder-eval into GitHub Actions

The user's request is: `$ARGUMENTS`

Step 1 — Check the repository

Find the repository's task tree by following `${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md`, and check whether `.github/workflows/` exists. The paths you resolve here become `args:` entries in the workflow in step 3 — that input is written from discovery, never from a fixed guess.

If there is no `.github/` directory at all, say that this skill targets GitHub Actions and stop — do not invent an equivalent for another CI system unless the user asks.

If a workflow already runs coder-eval (grep the workflows for `coder_eval`), do not add a second one. Show what is there and offer to update it.

Step 2 — Choose the trigger

Ask, or infer from the request:

• **On pull request** — gate changes to the tasks or to whatever they exercise.
• **On a schedule** — the skill-drift case: re-run the suite weekly against the current

model so a skill that quietly stops triggering surfaces before users hit it. This is the trigger most repositories actually want, and the one they forget. If the suite is an activation suite, the environment note in step 3 is **not optional** for this trigger — without it the scheduled run reports total drift every week regardless of whether anything drifted.

• **Both**, which is fine — one workflow, two `on:` keys.

Step 3 — Emit the workflow

The composite action installs the `coder-eval` CLI and nothing else: it is agent-agnostic and installs **no coding-agent runtime**. A task using the default `claude-code` agent therefore needs Node plus the Claude CLI provided by the job first, or the run dies on a missing `claude` binary. There is no Marketplace install step for the action itself, but those two prerequisite steps are not optional.

name: Coder Eval

on:
  pull_request:
  schedule:
    - cron: "0 6 * * 1"   # Mondays 06:00 UTC — catches model/skill drift

# Least privilege: this job runs agent-generated code, so it gets no write scope.
permissions:
  contents: read

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v6
        with:
          # Do not leave a credentialed .git/config in a workspace where
          # agent-generated code runs.
          persist-credentials: false

      # The action installs no coding-agent runtime — provide it here.
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - run: npm install -g @anthropic-ai/claude-code

      - id: eval
        uses: UiPath/coder_eval@v0
        with:
          run-dir: runs/ci
          args: |
            tasks/**/*.yaml
            --model
            claude-haiku-4-5-20251001
          env: |
            ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}

      - if: always()
        run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY"

Adjust the model and the cron to the repository. Pin the action at `@v0`, the moving major tag. Then work through the four things the snippet cannot guess.

Note the shape of `args:`: the action promotes **none** of `coder-eval run`'s flags to a named input, so task paths and flags all go there, one argument per line, with a flag and its value on separate lines. There is no `tasks:`, `tags:` or `model:` input.

The task paths — from discovery

The value above is a placeholder for whatever step 1 discovered. `args:` entries are handed to the CLI **verbatim**: no word splitting, no pathname expansion. The CLI expands the globs itself, which makes this simpler than it looks:

• **`**` works.** `tasks/**/*.yaml` is genuinely recursive, so one pattern covers a tree

of any depth. No per-depth ladder, no `globstar` caveat.

• **A glob matching nothing fails loudly** with `No task files found!` and exit 1, rather

than reaching the CLI as a literal path.

• **One path per line.** Two globs are two lines, not one space-separated string, which

would arrive as a single malformed argument.

So emit what step 1 found, one entry per line:

args: |
  tests/tasks/**/*.yaml

`version:` — conditional on the repository's pin

If the repository pins a coder-eval version, **pass it** and say why: the gate should run the CLI the repo is authored against, not whichever release the action defaults to. If there is no pin, omit the input and let the action's default track the matching release. `${CLAUDE_PLUGIN_ROOT}/reference/cli-setup.md` covers how to find a pin — and passing one explicitly is right even when it happens to match today's default, because it is self-documenting.

The experiment, if the suite runs through one

If the repository's suite resolves through an experiment, the workflow must pass it in `args` — two lines, and with the discovered path, not the illustrative one below:

args: |
  tests/tasks/**/*.yaml
  -e
  tests/experiments/default.yaml

This matters rather than being tidy: an experiment usually supplies `agent:` config, so omitting it silently changes what the run measures — the gate and the local run stop being the same test. If the repository has **several** experiments, ask which one the gate should use; a CI gate quietly running the wrong experiment is precisely the failure this exists to prevent.

Environment — including the skill source, if the suite is an activation suite

If the resolved experiment or the tasks interpolate environment variables, pass them through the action's `env:` input alongside the credentials. Missing ones do not fail loudly — the run just measures the wrong thing.

One case is common enough to check for by name. An activation