/triage-ci-flake

name: triage-ci-flake
description: Use when CI tests fail on main branch after PR merge, when investigating flaky test failures, or when user provides a PR URL/number to aggregate all failing tests
allowed-tools: Write, Bash(date:*), Bash(mkdir -p *)

Triage CI Failure

Overview

Systematic workflow for triaging and fixing test failures in CI, especially flaky tests that pass locally but fail in CI. Tests that made it to `main` are usually flaky due to timing, bundling, or environment differences.

**CRITICAL RULE: You MUST run the reproduction workflow before proposing any fixes. No exceptions.**

When to Use

• CI test fails on `main` branch after PR was merged
• Test passes locally but fails in CI
• Test failure labeled as "flaky" or intermittent
• E2E or integration test timing out in CI only
• User provides a PR URL/number with failing checks

PR-Based Workflow (When PR URL/Number Provided)

When the user provides a PR URL (e.g., `https://github.com/payloadcms/payload/pull/16701`) or PR number:

Step 1: Fetch PR Status Checks

First, use `tool_search` with query "github pull request status checks" to load the GitHub tools.

Then use the `github-pull-request_pullRequestStatusChecks` tool to get all failing checks:

Tool: github-pull-request_pullRequestStatusChecks
Parameters:
  pullRequestNumber: <extracted from URL or provided>
  repo: { owner: "payloadcms", name: "payload" }

Step 2: Aggregate Failing Tests

Parse the status checks response and create a summary table:

| Suite | Check Status | Target URL | | --------------------- | ------------ | ---------- | | admin**e2e**list-view | failed | [link] | | plugin-import-export | failed | [link] |

For each failing check, the `context` field contains the test suite name (e.g., `admin__e2e__list-view`).

Step 3: Extract Failure Details from Each Check

For each failing check:

1. Visit the `targetUrl` to get detailed failure logs 2. Extract the specific test name and error message 3. Add to the aggregated failure list

Present a consolidated summary:

## Failing Tests Summary

### 1. admin**e2e**list-view (3/4)

- **Test**: "should use custom pagination limit"
- **Error**: Locator `.per-page__button` not found
- **File**: test/admin/e2e/list-view/e2e.spec.ts:1495

### 2. plugin-import-export

- **Test**: "should inherit limit from list view URL"
- **Error**: Locator `.per-page__button` not found
- **File**: test/plugin-import-export/e2e.spec.ts:150

Step 4: Identify Common Patterns

Look for patterns across failures:

• Same selector errors → likely a component change needing test updates
• Same test file → localized issue
• Different errors in same suite → may be test pollution

Step 5: Proceed with Triage

For each unique failure, follow the standard reproduction workflow below.

MANDATORY First Steps

**YOU MUST EXECUTE THESE COMMANDS. Reading code or analyzing logs does NOT count as reproduction.**

1. **Extract** suite name, test name, and error from CI logs 2. **EXECUTE**: Kill port 3000 to avoid conflicts 3. **EXECUTE**: `pnpm dev $SUITE_NAME` (use run_in_background=true) 4. **EXECUTE**: Wait for server to be ready (check with curl or sleep) 5. **EXECUTE**: Run the specific failing test with Playwright directly (npx playwright test test/TEST_SUITE_NAME/e2e.spec.ts:31:3 --headed -g "TEST_DESCRIPTION_TARGET_GOES_HERE") 6. **If test passes**, **EXECUTE**: `pnpm prepare-run-test-against-prod` 7. **EXECUTE**: `pnpm dev:prod $SUITE_NAME` and run test again

**Only after EXECUTING these commands and seeing their output** can you proceed to analysis and fixes.

**"Analysis from logs" is NOT reproduction. You must RUN the commands.**

Core Workflow

digraph triage_ci {
    "CI failure reported" [shape=box];
    "Extract details from CI logs" [shape=box];
    "Identify suite and test name" [shape=box];
    "Run dev server: pnpm dev $SUITE" [shape=box];
    "Run specific test by name" [shape=box];
    "Did test fail?" [shape=diamond];
    "Debug with dev code" [shape=box];
    "Run prepare-run-test-against-prod" [shape=box];
    "Run: pnpm dev:prod $SUITE" [shape=box];
    "Run specific test again" [shape=box];
    "Did test fail now?" [shape=diamond];
    "Debug bundling issue" [shape=box];
    "Unable to reproduce - check logs" [shape=box];
    "Fix and verify" [shape=box];

    "CI failure reported" -> "Extract details from CI logs";
    "Extract details from CI logs" -> "Identify suite and test name";
    "Identify suite and test name" -> "Run dev server: pnpm dev $SUITE";
    "Run dev server: pnpm dev $SUITE" -> "Run specific test by name";
    "Run specific test by name" -> "Did test fail?";
    "Did test fail?" -> "Debug with dev code" [label="yes"];
    "Did test fail?" -> "Run prepare-run-test-against-prod" [label="no"];
    "Run prepare-run-test-against-prod" -> "Run: pnpm dev:prod $SUITE";
    "Run: pnpm dev:prod $SUITE" -> "Run specific test again";
    "Run specific test again" -> "Did test fail now?";
    "Did test fail now?" -> "Debug bundling issue" [label="yes"];
    "Did test fail now?" -> "Unable to reproduce - check logs" [label="no"];
    "Debug with dev code" -> "Fix and verify";
    "Debug bundling issue" -> "Fix and verify";
}

Step-by-Step Process

1. Extract CI Details

From CI logs or GitHub Actions URL, identify:

• **Suite name**: Directory name (e.g., `i18n`, `fields`, `lexical`)
• **Test file**: Full path (e.g., `test/i18n/e2e.spec.ts`)
• **Test name**: Exact test description
• **Error message**: Full stack trace
• **Test type**: E2E (Playwright) or integration (Vitest)

2. Reproduce with Dev Code

**CRITICAL: Always run the specific test by name, not the full suite.**

**SERVER MANAGEMENT RULES:**

1. **ALWAYS kill all servers before starting a new one** 2. **NEVER assume ports are free** 3. **ALWAYS wait for server ready confirmation before running tests**

# ==============