/test-runner

name: test-runner
user-invocable: false
tags: [test, orchestrator, e2e, ux]
model: sonnet
model-preference: sonnet
model-preference-codex: gpt-5.4
model-preference-cursor: claude-sonnet-4-6
description: >
  Use this skill when orchestrating agentic end-to-end tests. Resolves target + profile, dispatches
  the right driver(s) (playwright for web today, peekaboo for macOS (issue #381)), invokes the ux-evaluator agent (opus, read-only) against
  driver artifacts, reconciles findings with the open issue tracker via
  scripts/lib/test-runner/issue-reconcile.mjs, and writes report.md +
  JSONL roll-up. Wraps upstream tools (no forks). Hard-gates Playwright MCP
  for browser drive (4× token cost vs CLI per Microsoft's own benchmark).

Test Runner Skill

> Project-instruction file resolution: `CLAUDE.md` and `AGENTS.md` (Codex CLI) are transparent aliases — see [skills/_shared/instruction-file-resolution.md](../_shared/instruction-file-resolution.md). Wherever this skill mentions `CLAUDE.md`, the alias rule applies.

Soul

Before anything else, read and internalize `soul.md` in this skill directory. It defines WHO you are — your role as an orchestrator, your delegation boundaries, and your non-negotiable constraints.

Phase 0: Bootstrap Gate

Read `skills/_shared/bootstrap-gate.md` and execute the gate check. If GATE_CLOSED, invoke `skills/bootstrap/SKILL.md` and wait for completion. If GATE_OPEN, continue to Phase 1.

<HARD-GATE> Do NOT proceed past Phase 0 if GATE_CLOSED. There is no bypass. Refer to `skills/_shared/bootstrap-gate.md` for the full HARD-GATE constraints. </HARD-GATE>

Phase 1: Read Session Config + Resolve Target / Profile

Read and parse Session Config per `skills/_shared/config-reading.md`. Store result as `$CONFIG`.

Test-runner specific fields (parse these specifically):

• `test-runner.default-profile` (default: `smoke`)
• `test-runner.retention-days` (default: `30`)
• `test-command`, `typecheck-command`, `lint-command` (used for context only — not driven here)

Target / Profile Resolution

Resolution order (first match wins):

1. **CLI argument** `--target <name> --profile <name>` (explicit, highest priority) 2. **Policy file lookup** — `.orchestrator/policy/test-profiles.json` by target name (if present) 3. **Convention-based detection** (marker files):

• `playwright.config.{ts,js}` present → target type `web`, dispatch `playwright-driver`
• `Package.swift` present → target type `mac`, dispatch `peekaboo-driver` (see `skills/peekaboo-driver/SKILL.md`)

4. **Fallback** → emit error and halt:

   Error: Cannot resolve target — provide --target or add .orchestrator/policy/test-profiles.json

Run ID

Generate a run ID immediately after target resolution:

import { makeRunId } from 'scripts/lib/test-runner/artifact-paths.mjs';
const runId = makeRunId(); // e.g. "your-target-app-1715688000123"

All artifact paths in subsequent phases derive from this run ID. Never use ad-hoc paths.

Status Report

After resolution, emit: `Test Runner: target=[name] profile=[name] run_id=[runId] driver=[driver]`

Phase 2: Driver Dispatch

Determine `${RUN_DIR}` from `artifact-paths.mjs:runDirPath(runId)` before dispatching any driver. All drivers write artifacts under `${RUN_DIR}/`.

--since Filtering (when `since_ref` is provided)

When `since_ref` is set (passed from the `/test --since <git-ref>` handoff contract):

1. Import and call `changedFilesSince(since_ref)` from `scripts/lib/discovery/helpers.mjs`. 2. If the helper throws (ref unresolvable), surface the error to the user and halt. 3. If the result is `[]` (no files changed since the ref), emit:

   No files changed since <since_ref>. Skipping test run.

and exit with status 0. Do NOT fall back to a full-repo test run. 4. If the result is a non-empty array, JSON-stringify it and set `TEST_CHANGED_FILES` in the driver subprocess environment (see driver invocations below). Driver-side filtering is deferred — drivers receive the env var but do not yet filter by it in this wave.

For each resolved driver:

Web (playwright-driver)

Dispatch via Bash per `skills/playwright-driver/SKILL.md`. Pass `${RUN_DIR}` so the driver writes all artifacts (screenshots, AX dumps, HAR) under it.

# Example invocation shape (exact flags defined by playwright-driver SKILL.md)
TEST_CHANGED_FILES="${CHANGED_FILES_JSON}" node scripts/lib/playwright-driver/runner.mjs \
  --run-dir "${RUN_DIR}" \
  --profile "${PROFILE}" \
  --target "${TARGET}"

Where `${CHANGED_FILES_JSON}` is `JSON.stringify(changedFiles)` when `--since` was provided, or an empty string otherwise.

Capture exit code. A non-zero exit from Playwright means test failures — these become findings for the UX evaluator. They are NOT a fatal error for the orchestrator. Continue to Phase 3 regardless of exit code.

Log: `playwright-driver exited [code] — [N] test files captured under ${RUN_DIR}`

macOS (peekaboo-driver)

> See `skills/peekaboo-driver/SKILL.md` for the full dispatch contract, permission probe, and artifact layout.

**Pre-dispatch platform check:** The driver's Phase 1 gate handles the platform and version checks (`darwin` + macOS 15.0+) and exits 0 (non-fatal skip) on incompatible systems. The orchestrator does not need to replicate these checks.

**Permission probe:** The driver runs its own Phase 2 permission probe via `peekaboo permissions status --json`. If required permissions (Screen Recording, Accessibility) are not granted, the driver surfaces an AUQ and exits 2 on failure. The orchestrator treats exit 2 as a driver-framework error, not a test failure.

**Invocation:**

# All inputs via environment variables — no positional arguments
RUN_DIR="${RUN_DIR}" TARGET="${TARGET}" PROFILE="${PROFILE}" bash skills/peekaboo-driver/SKILL.md

Any bash-driver shim or report helper you write for this dispatch is subject to `.claude/rules/bash-harness-pitfalls.md` (`grep -c` double-p