/test-runner

name: test-runner
user-invocable: false
tags: [test, orchestrator, e2e, ux]
model: sonnet
model-preference: sonnet
model-preference-codex: gpt-5.4
model-preference-cursor: claude-sonnet-4-6
description: >
  Use this skill when orchestrating agentic end-to-end tests. Resolves target + profile, dispatches
  the right driver(s) (playwright for web today, peekaboo for macOS (issue #381)), invokes the ux-evaluator agent (opus, read-only) against
  driver artifacts, reconciles findings with the open issue tracker via
  scripts/lib/test-runner/issue-reconcile.mjs, and writes report.md +
  JSONL roll-up. Wraps upstream tools (no forks). Hard-gates Playwright MCP
  for browser drive (4× token cost vs CLI per Microsoft's own benchmark).

Test Runner Skill

> Project-instruction file resolution: `CLAUDE.md` and `AGENTS.md` (Codex CLI) are transparent aliases — see [skills/_shared/instruction-file-resolution.md](../_shared/instruction-file-resolution.md). Wherever this skill mentions `CLAUDE.md`, the alias rule applies.

Soul

Before anything else, read and internalize `soul.md` in this skill directory. It defines WHO you are — your role as an orchestrator, your delegation boundaries, and your non-negotiable constraints.

Phase 0: Bootstrap Gate

Read `skills/_shared/bootstrap-gate.md` and execute the gate check. If GATE_CLOSED, invoke `skills/bootstrap/SKILL.md` and wait for completion. If GATE_OPEN, continue to Phase 1.

<HARD-GATE> Do NOT proceed past Phase 0 if GATE_CLOSED. There is no bypass. Refer to `skills/_shared/bootstrap-gate.md` for the full HARD-GATE constraints. </HARD-GATE>

Phase 1: Read Session Config + Resolve Target / Profile

Read and parse Session Config per `skills/_shared/config-reading.md`. Store result as `$CONFIG`.

Test-runner specific fields (parse these specifically):

• `test-runner.default-profile` (default: `smoke`)
• `test-runner.retention-days` (default: `30`)
• `test-command`, `typecheck-command`, `lint-command` (used for context only — not driven here)

Target / Profile Resolution

Resolution order (first match wins):

1. **CLI argument** `--target <name> --profile <name>` (explicit, highest priority) 2. **Policy file lookup** — `.orchestrator/policy/test-profiles.json` by target name (if present) 3. **Convention-based detection** (marker files):

• `playwright.config.{ts,js}` present → target type `web`, dispatch `playwright-driver`
• `Package.swift` present → target type `mac`, dispatch `peekaboo-driver` (see `skills/peekaboo-driver/SKILL.md`)

4. **Fallback** → emit error and halt:

   Error: Cannot resolve target — provide --target or add .orchestrator/policy/test-profiles.json

Rubric and Target Label

Resolve the rubric and requested target right after the profile; verify the target against Phase 2 capture evidence before Phase 3 passes it to the evaluator.

**`RUBRIC`** — the selected profile's `rubric` field:

import { loadProfiles, getProfile } from 'scripts/lib/profiles/registry.mjs';
const reg = await loadProfiles();                      // .orchestrator/policy/test-profiles.json
const hit = reg.ok ? getProfile(reg.profiles, PROFILE) : { ok: false };
const RUBRIC = hit.ok ? hit.profile.rubric : 'skills/test-runner/rubric-v2.md';

`getProfile()` already applies the schema default (`scripts/lib/profiles/schema.mjs`, rubric-v2) when a profile names no rubric, so a pin is honoured and an unpinned profile gets v2. No registry file or no matching profile (convention-based detection) → rubric-v2. The mac profiles pin `rubric-v1.md` on purpose: peekaboo writes `ax-snapshots/<scenario>.json`, no 1440/390 viewports, so the v2 human checks could never complete there.

**`TARGET_KIND` / `TARGET_URL`** — what the run actually looked at (rubric-v2 Hard rule 4):

• Consume `target_kind` and `target_url` from the `/test` handoff. A recognized `--target-kind` (`live`, `production-build`, `dev-server`) overrides an operator statement; without that flag, use an explicit statement for this run, otherwise `dev-server`. An invalid or missing flag value forces the pair `dev-server unknown` even if a URL was supplied. No profile field upgrades the kind.
• `TARGET_URL` is `target_url` from `--target-url` first, then the URL the operator named, then the web target's `playwright.config.{ts,js}` `use.baseURL` (read the file, never execute it); for a macOS target use the app path. No value, or an explicitly missing flag value, is the literal `unknown`. Use the resolved URL for browser navigation as well as labelling; a label must describe the target actually driven.
• A `localhost` / `127.0.0.1` / branch-preview URL is always `dev-server`, whatever was stated or flagged.
• Unknown is never upgraded: the evaluator gets `dev-server unknown`, and its report header says `DEV-SERVER RUN`.
• For web targets, treat `TARGET_KIND` as requested until capture evidence confirms `${TARGET_URL}` was actually driven. Grant `LIVE RUN` / `PRODUCTION-BUILD RUN` only when the per-page visited-URL record from Phase 2 supports that target. Missing records or any visited origin differing from `${TARGET_URL}`'s origin force `TARGET_KIND = dev-server`; name the requested URL, observed URLs (or missing evidence), and mismatch in the report. Use the observed URL for `TARGET_URL`, or `unknown` if absent. After verification, set `TARGET_HEADER` to `LIVE RUN`, `PRODUCTION-BUILD RUN`, or `DEV-SERVER RUN — not the live state; no verdict about what users see.` respectively; keep `Target: ${TARGET_KIND} ${TARGET_URL}` consistent with that result.

Run ID

Generate a run ID immediately after target resolution:

import { makeRunId } from 'scripts/lib/test-runner/artifact-paths.mjs';
const runId = makeRunId(); // e.g. "your-target-app-1715688000123"

All artifact paths in subsequent phases derive from this run ID. Never use ad-hoc paths.

Status Report

After resolution, emit: `Test Runner: target=[name] profile=[name] run_id=[runId] driver=[drive