/pr-test-automation

name: pr-test-automation
description: Run first-pass automated manual testing for PRs that are reviewed or nearly ready to merge. Use when the user asks to test a PR/branch/worktree, validate product flows, exercise browser or CLI workflows, map changed UI journeys with screenshots, verify analytics/webhooks/payments/email/SMS behavior through connected tools, or produce manual QA notes before human testing.

PR Test Automation

Overview

Validate as much of a PR as possible with local services, browser automation, CLIs, logs, and product connectors before the user does final manual testing. Treat this as a first-pass QA workflow: prove what works with evidence, identify what still needs a human, and preserve a reproducible trail.

Workflow

1. Confirm the test target:

• Identify PR numbers, branches, worktrees, related companion PRs, and whether the user allowed rebasing/syncing.
• Check `git status`, current branch, remotes, and whether unrelated local changes exist.
• Inspect PR descriptions/review notes when they define required manual flows.

2. Check tools and authentication up front:

• Verify required CLIs and connectors before starting long tests: `gh auth status`, `stripe --version` / active listener state, Docker status, PostHog/GitHub/Gmail connectors, cloud CLIs, or app-specific CLIs.
• Discover verification tools before assuming a manual check is required. If Composio is available, use `composio search` to find candidate inbox, SMS/phone, payment, CRM, support, or provider-log tools, then inspect schemas with `--get-schema` before executing.
• Prefer connected app tools for product data verification. Do not assume credentials are current.
• Use test-mode accounts, test keys, local containers, and staging-safe endpoints unless the user explicitly asks for production verification.

3. Prepare the environment:

• Install dependencies only where needed and report anything that changes lockfiles.
• Start required dev servers or confirm existing sessions, ports, and mounted worktrees.
• For companion PRs, test the combined state in the worktree/container that actually serves the code.
• Avoid leaving duplicate background listeners or servers. List and clean up only processes started for the test.

4. Build the automated test path:

• Use Playwright when browser behavior matters. If the repo lacks Playwright, install it in a temporary directory rather than polluting the repo.
• Use stable, user-visible selectors first: labels, placeholders, button text, URLs, and route state.
• Generate unique short test identities and attribution markers such as `agent-e2e-<timestamp>`.
• When UI changes are in scope, map each touched surface area and user journey to screenshots in a temporary, easy-to-observe folder such as `tmp/pr-<number>-qa/` or `tmp/<branch>-qa/`. Use ordered filenames that describe the journey step, such as `01-signup-account.png` and `02-dropdown-expanded.png`.
• Capture meaningful UI states, not only final pages: empty/default, filled/selected, expanded menus, modals, validation errors, loading/success states, and at least one narrow viewport when responsive layout is likely affected.
• Reuse the same screenshot artifact pattern for local/dev validation and, when the user asks for post-merge production verification, for production paths. Keep local and production artifacts separated by folder or filename.
• When a journey is driven by a scriptable browser driver, record it as a video alongside the stills. One video per journey, recorded at the driver level (e.g. Playwright's `recordVideo`) so it is a free byproduct of the drive, not a second pass. Keep the driver's native format (WebM from browser, MP4 from simulator). Videos complement stills, never replace them: per-step captures remain the frame-addressable evidence, the video is the continuity check. Where ffmpeg is available, scan for blank-frame bands (`ffprobe -f lavfi "movie=<video>,fps=5,signalstats" -show_entries frame=pts_time -show_entries frame_tags=lavfi.signalstats.YAVG`; YAVG ~235 is blank white) and report layout jumps, white flashes, or dead time as findings with timestamp ranges.
• Prefer the app's built-in test/simulation path for external effects: local inboxes, Mailhog-style UIs, fake SMS numbers, test OTP logs, sandbox payment modes, webhook listeners, or provider test keys.
• Parse local email/SMS verification links or codes from container logs when the local environment emits them.
• Add small human-paced waits around analytics or step-transition tests so effects and batched events have time to fire in the same order a user would experience.

5. Verify externally, not just locally:

• Network requests prove the browser tried to send data; connector/API queries prove the product received it.
• When simulation is unavailable, use connected recipient/provider readback: Gmail/Outlook/IMAP or email-service activity for email; Twilio/Dialpad/OpenPhone/Google Voice/test-number services or provider logs for SMS/voice; Stripe/provider dashboards for payments.
• Query by the unique marker, test email, phone number, org ID, subscription ID, webhook event ID, request ID, or other stable test value.
• For webhooks, confirm both CLI/listener output and backend logs, then verify downstream data.
• For analytics dashboards, query the exact project and call out the date range and filters used.

6. Report results:

• Return what passed, remaining uncertainty, the next check, and up to three improvements grounded in testing friction.

Open with a verdict, then the evidence. The structure:

   Verdict: <all-proven | partial | blocked-env | blocked-auth | product-bug-found>

   | Journey / Check | Result | Evidence |
   |-----------------|--------|----------|
   | <flow or check> | Pass / Fail / Blocked / Left to human | <quoted output, screenshot ref, connector readback> |

   Skipped (with rationale):
   - <item>: <