/railway-test

name: railway-test
description: Triggered when the user invokes /railway-test or $railway-test, asks to test a pull request on Railway, or requests browser-based exact-head preview QA and pull-request evidence before merging.

Railway Test

Test the behavior the PR claims to change against the exact deployed PR head. Derive the browser plan from the PR instead of running a fixed chat scenario.

Inputs and secrets

Resolve from the request or current repository:

• PR number or URL. If omitted, use `gh pr view`.
• Repository. Default to `nearai/ironclaw`; the preview URL fallback is

repo-specific, so an explicit repository must be supplied when testing another repo.

• Preview URL. Resolve it from the request or Railway status. For

`nearai/ironclaw`, fall back to `https://ironclaw-ironclaw-pr-<PR>.up.railway.app`.

• Preview bearer token. Reuse an authenticated browser session when possible;

otherwise request it from the user.

Never persist the bearer in the skill, repository, scripts, shell history, URLs, PR text, logs, screenshots, or final response. Never print or repeat it. Type it only into the preview login UI. Do not inspect or return browser storage values containing it.

Workflow

1. Understand the PR before opening Railway

Read repository guidance relevant to testing. For Ironclaw, read `docs/internal/testing-playbook.md` before selecting tests.

Inspect:

gh pr view <PR> --repo <REPO> \
  --json number,title,body,url,baseRefName,headRefName,headRefOid,files
gh pr diff <PR> --repo <REPO>

Identify:

• the user-visible behavior promised by the PR;
• affected routes, screens, roles, and state;
• the exact regression or before/after claim;
• important success, validation, permission, and error paths;
• dependencies on external services, model behavior, or persisted data.

Write a compact Given/When/Then test matrix before interacting with the preview. Give every row an `Acceptance` classification:

• **Required**: directly proves an explicit user request, PR acceptance

criterion, or the exact regression claim.

• **Supplemental**: deployment health, adjacent behavior, or a useful proxy

that does not itself prove the changed contract.

Use [references/test-recipes.md](references/test-recipes.md) to choose only relevant cases. Do not default to chat or streaming tests. Never reclassify a required case as supplemental because the preview lacks its provider, model, credentials, role, feature flag, or test data; that makes the run BLOCKED.

Mandatory status gate

Compute the overall heading from required cases only:

1. **FAIL** if any required case was executed against the intended contract and the observed behavior contradicted the acceptance claim. 2. Otherwise **BLOCKED** if any required case was not executed, was inconclusive, or used a substitute provider, model, route, role, tool, state transition, or pre-existing object instead of the intended one. 3. **PASS** only if every required case was executed against the intended contract and passed.

Passing supplemental cases, a healthy deployment, local tests, recorded fixtures, or a nearest-caller check can add evidence but can never upgrade FAIL or BLOCKED to PASS. If no required case can run, stop after collecting bounded diagnostics and publish BLOCKED; do not invent a proxy journey merely to produce a browser PASS.

2. Confirm the intended build is live

Run (resolve the path against this skill's `scripts/` directory):

scripts/preview_state.sh <PR> [REPO] [PREVIEW_URL]

Record `head_sha`, `railway_state` (`success`, `pending`, `failure`, `error`, `missing`, or `query_failed`), `railway_details`, and `asset`. The script queries the commit-scoped status and check-run endpoints, so the result is bound to the tested head SHA — never a stale build. Poll every 30–45 seconds, up to 20 attempts (roughly 15 minutes), until `railway_state` reports `success`. Send a concise progress update at least once per minute.

If the exact-head Railway status is still `pending`, `missing`, or `query_failed` after the attempt limit, stop polling and report BLOCKED with the latest diagnostics; never test a partial build. A `failure` or `error` status is BLOCKED as well until the intended build is live.

For frontend changes, compare the asset hash with the pre-deploy baseline. For backend-only changes, an unchanged asset is acceptable after the head-specific Railway check passes. Do not test a stale build.

If Railway fails, inspect the linked check/deployment and stop browser testing until the intended build is live.

The PR head can move while this skill runs. Re-read `headRefOid` immediately before opening the preview (step 3) and again immediately before publishing (step 7). If either read differs from `head_sha`, restart build confirmation for the new head, or report BLOCKED for the stale build; never run or publish browser evidence against a head you did not confirm.

3. Open and authenticate the preview

Re-read `headRefOid` and confirm it still equals the `head_sha` recorded in step 2 before opening the preview; if it changed, return to step 2 for the new head instead of testing a stale build.

Select the browser driver available in the current agent harness:

• If `browser:control-in-app-browser` is available, read and follow it and use

the in-app browser.

• Otherwise use the harness's supported browser automation capability (for

example a browser tool, Playwright integration, or Chrome integration) that can inspect rendered DOM and interact through visible controls.

• If no browser automation capability is available, mark every required

browser case BLOCKED. HTTP requests, local tests, or source inspection may be supplemental evidence, but they do not constitute browser acceptance.

Announce which driver is opening the preview and that the supplied token will be used only for UI login. Open the route most relevant to the PR; use `/chat` only for chat c