/smoke-test

name: smoke-test
description: "Run smoke tests against a deployed or local app based on your git diff. Each test uses Skyvern browser tools (navigate, act, validate, screenshot) with Chrome DevTools MCP as fallback. Posts screenshot evidence as PR comments."

Smoke Test — CI-Oriented Validation via Skyvern Browser Tools

Read the diff, classify changes, start the app, and run targeted smoke tests via Skyvern browser tools with Chrome DevTools MCP as fallback.

<!-- NOTE: .agents/skills/smoke-test/SKILL.md is the repository canonical source. Keep the synchronized bundled copy in sync with it: skyvern/cli/skills/smoke-test/SKILL.md Steps 1-4 are copied from .agents/skills/qa/SKILL.md. If you fix bugs in /qa's diff-reading, classification, or app startup, mirror those fixes here. -->

You changed code. This skill reads the diff, generates targeted smoke tests, and runs each one via Skyvern browser tools - navigate, act, validate, screenshot. If Skyvern tools are unavailable (no Skyvern MCP), fall back to Chrome DevTools MCP, which connects to a real Chrome instance and supports authenticated sessions. It is /qa's CI companion: same diff-reading, same classification, same app startup, formatted for CI output and PR comments with screenshot evidence on every test.

Quick Start

/smoke-test                              # Diff-driven, auto-detect everything
/smoke-test https://staging.example.com  # Explicit app URL
/smoke-test -- focus on the settings page
/smoke-test --pr 11488                   # Target a specific PR

How It Works

1. Read git diff (reused from /qa) 2. Classify changes → identify testable surfaces (reused from /qa) 3. Choose validation strategy (reused from /qa) 4. Pick browser backend (Chrome DevTools MCP or Skyvern) 5. Handle auth (connect to authenticated session or bypass) 6. Start the app if needed (reused from /qa) 7. Generate 3-8 smoke test cases as action sequences (happy paths only) 8. Run each test via browser tools: navigate → act → validate → screenshot 9. Collect results with screenshot evidence 10. Report with embedded screenshots 11. Post to PR with images

Step 1: Understand the Changes

Get the diff

# What files changed?
git diff --name-only HEAD~1     # vs last commit (if changes are committed)
git diff --name-only            # vs working tree (if uncommitted)

# Full diff for context
git diff HEAD~1                 # or git diff for uncommitted

Pick whichever diff has content. If both are empty, there is nothing diff-driven to test.

Read the changed files

Read the full contents of every changed file that affects behavior:

• Frontend files: `.tsx`, `.jsx`, `.ts`, `.js`, `.css`, `.html`
• Backend/API files: routes, controllers, request/response schemas, serializers, handlers
• Backend-internal files: services, workers, business logic, validators, data-layer code
• Tests that changed alongside the implementation

Look for:

• route paths and page entry points
• component names, visible text, forms, buttons, error states
• API endpoints, request params, response fields, auth requirements
• validation logic, branching behavior, feature flags, empty states
• tests that describe the expected behavior

Step 2: Classify the Diff

| Mode | Trigger | Primary validation | |------|---------|--------------------| | Frontend/browser | UI/routes/components/styles changed | Browser smoke tests against the dev server | | Backend API | Route handlers, request/response schemas, or externally visible API behavior changed | Start backend locally and run smoke tests against changed endpoints | | Backend-internal | Services/workers/business logic changed without public API surface changes | Repo-native fast checks plus targeted tests | | Mixed | Frontend/browser and backend changed together | Backend validation first, then frontend smoke tests |

Use these rules:

• If both frontend/browser and backend changed, treat it as `Mixed`.
• If only backend internals changed, do not invent unrelated browser tests or random API calls.
• If a backend change might affect the public contract, inspect routes, schemas, and tests before choosing `backend-internal`.
• If the diff is mostly documentation or comments, keep testing lightweight and report that no behavioral validation was warranted.

Step 3: Choose the Validation Strategy

Frontend/browser mode

Run smoke tests via Skyvern browser tools against the dev server. Validate the specific UI changes plus 1-2 adjacent regression checks.

Backend API mode

Use the repo's documented local startup and auth instructions, start the backend if needed, identify the changed endpoint(s), and run smoke tests that validate the changed contract.

Backend-internal mode

Run the repo's fast verification commands first, then targeted unit/integration/scenario tests for the changed logic. Only run smoke tests if the change affects exposed behavior.

Mixed mode

Validate the backend first, then run frontend smoke tests against the flow that depends on it. If the backend contract is broken, frontend results are not trustworthy.

Step 3b: Pick Browser Backend

Try Skyvern browser tools first. Fall back to Chrome DevTools MCP if Skyvern MCP is unavailable.

Skyvern browser tools (primary)

Use `skyvern_browser_session_create`, `skyvern_navigate`, `skyvern_act`, `skyvern_validate`, `skyvern_screenshot`. Works in CI and locally. Create a `local=true` session to reach localhost.

Chrome DevTools MCP (fallback)

If Skyvern MCP tools are not available, invoke the `/chrome-devtools` skill to learn the tool API. Key tools:

• `list_pages` / `new_page` / `navigate_page` - page management
• `take_screenshot` - capture viewport (save to a path within workspace roots)
• `take_snapshot` - get page structure with element `uid`s for interaction
• `click` / `fill` / `press_key` - interact with elements by `uid`
• `evaluate_script` - run JS for health gates and da