/fixing-develop-ci

name: fixing-develop-ci
description: Triages failed CI runs for a commit merged into develop and resolves them. Reruns the failed jobs when the failure is transient (flaky test, network or runner hiccup), fixes a real breakage on develop and opens a PR, or reports failures that need a maintainer, such as missing secrets or an external outage. Use when push CI on develop fails, when asked to investigate or fix a red develop commit, or when the ai-fix-develop-ci workflow runs.

Fixing CI on develop

Pushes to `develop` run the PR CI workflows, plus Performance Suite and Load Testing, which PRs run only with a label. A merged PR can still turn `develop` red. Most failures are transient: outages, network errors, and flaky tests. Real breakages usually come from a PR whose last CI run predates a change on `develop`, such as a new lint rule, a theme change that invalidates new snapshots, or a file length limit. Others come from a workflow the PR did not run, or from a runner image or external service change.

Most workflows cancel the run for an older commit when a newer one lands, so the newest run of each workflow decides whether `develop` is green. PR wheel-size and coverage comments compare against the latest green `develop` run, so while `develop` is red they compare against an older commit.

The goal is a green `develop` with the least work: rerun what was transient, fix what is broken, and hand the rest to a maintainer.

Inputs

A commit SHA on `develop`, or the URL of one of its failed runs. Without one, use the newest `develop` commit with a failed push run.

Workflow

- [ ] Collect the failed runs, jobs, and first errors
- [ ] Compare against the merged PR and history
- [ ] Classify: rerun, fix, or report
- [ ] Act on the decision
- [ ] Write the summary

1. Collect the failure

• List the commit's push runs (`gh run list --commit <full-sha> --event push`; a short SHA matches nothing) and pick the failed ones.
• For each failed run, list the failed jobs and steps of its latest attempt, and find the first real error in each failed job. Ignore cascading failures and skipped jobs.
• Start with the check-run annotations (`gh api repos/streamlit/streamlit/check-runs/<job-id>/annotations`). They usually name the error and cost less than logs. Read the failed logs (`gh run view <id> --log-failed`) when the annotations are not enough.
• A failed run whose jobs all show `cancelled`, with a "not acquired by Runner" annotation, is a runner outage.
• Note which variants failed: Python version, browser, OS, test file.
• For Playwright failures, the run uploads a `playwright_test_results_*` artifact with traces and a `snapshot-updates/` folder holding the CI-rendered images. Download artifacts into `work-tmp/`, never into tracked folders.

2. Compare against the merged PR and history

• Find the PR the commit came from (`gh pr list --state merged --search <sha>`). Check what it changed and whether its own CI passed. A failure its CI did not show usually comes from a PR merged after that CI ran, or from an external change such as a new tool or dependency release.
• Check newer `develop` commits. If a newer push run of the same workflow passed, the failure no longer affects `develop`. If one is still running, choose **report**: its result decides, and its own triage handles a failure.
• Check the last ~10 push runs of the failing workflow on `develop` and recent nightly runs for the same error. A deterministic failure repeats and usually traces back to one commit.
• For a test that might be flaky, check for `@pytest.mark.flaky` and the output of `uv run scripts/fetch_flaky_tests.py`.

3. Classify

| Decision | Signals | |----------|---------| | **rerun** (transient) | Network or registry errors, including in `Set up job` or `Setup virtual env` (yarn, uv, pip, apt, a `curl` 5xx while downloading tools, GitHub API 5xx, `ECONNRESET`). A runner that was lost, shut down, or never acquired. Artifact upload or download errors, including a 403 on finalize. Job timeouts with no test failure. A brief outage of an external service such as CodeQL ("No server is currently available"), Snowflake, or S3. A single test failing on one browser or Python version that passes elsewhere in the run and in other recent runs. | | **fix** (real breakage) | The same failure across browsers, Python versions, matrix jobs, or attempts. Lint, format, type, pre-commit, or build errors, including `uv lock --check`, Node version, or protobuf errors in `Setup virtual env`. Snapshot mismatches in every browser, or on files touched by a visual or theme PR merged shortly before. A flaky test that also failed on the previous non-cancelled runs. Many tests failing with OS-level errors (time zones, missing shared libraries) after a runner image update. The failure reproduces on current `develop`. Tests that depend on the current date and break on a specific day also count. | | **report** (needs a maintainer) | Missing or invalid secrets (for example, a Snowflake login error), missing permissions, or rate limits on the bot account. An external outage that will not clear soon. A fix that needs a `.github/workflows/` change. The rerun limit is reached and no clear fix exists. A newer `develop` run of the same workflow passed or is still running, or `develop` or an open PR already fixes the failure. Only reverting the introducing PR would fix it. |

Keep these in mind:

• Playwright retries a failing test once (`--reruns 1` in `playwright.yml`), except failures whose message contains `Missing snapshot` (`--rerun-except` in `e2e_playwright/pytest.ini`). A retried failure in the log failed twice in that attempt. It can still be flaky when it is isolated to one browser and passes in other runs.
• If any failure is a real breakage, choose **fix**. Merging the fix runs every workflow again, so rerunning the other failed runs adds nothing.
• Allow at most 3 attempts per run (the original plus 2 reruns). After that