/fix-flaky-tests

name: fix-flaky-tests
description: Investigate and fix flaky or intermittently-failing tests in any framework and CI system — detects the framework, CI provider, and app profile, then applies the matching classification and fix patterns. Triggers on a flaky-test issue or CI URL, a test that "passes on retry" or "fails in CI but passes locally", a bot that skipped a test, or a request to reopen or dispute the closure or diagnosis of a flaky-test issue (for that case open the issue first; its title or label confirms scope even when the prompt has no flaky-test keywords).
allowed-tools: Bash Read Grep Glob Edit Write Skill

Fix Flaky Tests

Reference guide for investigating and fixing flaky tests across frameworks (RSpec, Jest, pytest, Go test, …), CI systems (Buildkite, CircleCI, GitHub Actions, …), and apps. The methodology is universal; framework idioms, CI log-fetch procedures, and app-specific flake catalogues are loaded on demand via **progressive discovery**. Compose the workflow to the situation rather than following rigid steps.

<HARD-RULES>

• NEVER skip a test (xit, skip, pending, `t.Skip`, `.skip`, etc.) as a fix
• NEVER fabricate a root cause — if you can't identify it, say so
• NEVER propose a diagnosis-driven fix without the actual CI error message (exception + backtrace — fetched from the CI logs OR already pasted in the prompt / by the user). The cheap fast exits are the sole exception: already-fixed, bot-skip revert, and broken-not-flaky resolve an issue from git/PR history without a CI error and may open a PR. This rule governs the real investigation that follows a failed fast-exit.
• NEVER proceed with a real investigation if the CI error is not in hand. **"In hand"** = the exception + backtrace is present in the prompt OR was retrieved by a tool. A broken log fetch does NOT mean "no error" when the caller already supplied it in the prompt — check the prompt first, and if the error is there, proceed normally (a headless run with its CI access down but the error pasted is a valid investigation). Only when the error is genuinely absent AND cannot be fetched: **interactive caller** (a user can answer) — stop and ask for it; **headless caller** — no interactive user to answer (an automated or scheduled invocation with nobody to ask; if unsure, assume headless) — ABORT and return the **CI-Logs-Unavailable Abort** (see HARD GATE below). Never downgrade to a guess or a "best-effort" hypothesis; there is no "reduced confidence" mode.
• NEVER run a test locally, for any reason — not `bundle exec rspec`, `script/test`, `jest`, `pytest`, `go test`, or any equivalent. Not to reproduce, not to confirm a mechanism, not to verify a fix. This holds for every flake category, including seed-reproducible state-poisoning/ordering, and having a local test environment available changes nothing. Diagnose from the CI error and source reading. Whether a replay can be staged on CI at all is a property of the pipeline, not a universal — some let you pin a seed or re-run a named shard; others re-shard on every build so there is nowhere to replay one. Read the discovered provider and profile files before concluding either way, and work from whatever evidence they name. The ban above is on running the test *here*; it says nothing about what CI can be asked to do.
• NEVER assume infrastructure flakiness without build-wide evidence
• Only fix if you can identify the root cause with HIGH confidence
• NEVER claim a fix is complete until CI is green — a "fix" that fails CI is not a fix. Monitor the PR build, investigate failures, and iterate until it passes.
• Verification of a proposed fix must come from a green CI build, never from local runs. Local runs cannot replicate CI's real backing services, test-isolation model, or cross-test ordering, so a local pass means almost nothing.

</HARD-RULES>

Required Input

One of:

• A bug-tracker issue link about a flaky test
• A test file path
• A CI build/job URL with a failing run
• An advisory question about a flaky test (reproduction strategy, verification, noise interpretation)
• A request to address, reopen, or dispute a *closed* flaky-test issue ("this was mis-identified / wrongly closed")

**Disputed or already-closed issues — re-derive, don't trust the close.** Treat any existing investigation or closing comment as a hypothesis, not a finding — re-establish the root cause from the actual CI logs (the HARD GATE below applies regardless of what the comment claims). Closing comments are frequently plausible-but-wrong: they cite a mechanism that doesn't match the real backtrace, or apply the infrastructure fast-exit ("a shared dependency was briefly down → no code fix") to a build where only **one** test failed. A single-example failure is the opposite of build-wide infra evidence — it usually means that one test is uniquely fragile: e.g. a strict assertion on a process-global sink (an error reporter, a metrics client) with no pass-through fallback, tripped by a benign *handled* report emitted by unrelated setup during the same example. That is a fixable code-side bug owned by the test's source team, not an infra ticket. If you reopen, correct the routing to that owning team.

Discover the Environment (do this first)

Before classifying anything, detect the stack so the right knowledge loads. Run the detection in **`references/discovery.md`** — it covers three tiers:

1. **Test framework** (from `Gemfile`/`package.json`/`pyproject.toml`/`go.mod` + test dirs) → load `references/frameworks/<framework>.md`. Only `rspec.md` ships fully fleshed; others fall back to `references/classification-generic.md` + `frameworks/_template.md`. 2. **CI provider** (from `.buildkite/`/`.circleci/`/`.github/workflows/`) → load `references/ci/<provider>.md` for how to fetch logs. Only `buildkite.md` ships fully fleshed; others use `ci/_template.md`, but the HARD GATE below still applies. 3. **App profile** (from `git remote get-url origin`) → load `refe