/fixing-nightly

name: fixing-nightly
description: Triages a failed Nightly Build run and resolves it. Reruns the failed jobs when the failure is transient (flaky test, network or runner hiccup), fixes a real breakage on develop and opens a PR, or reports failures that need a maintainer, such as PyPI publishing errors. Use when the nightly build fails, when asked to investigate or fix the nightly, or when the ai-fix-nightly workflow runs.

Fixing nightly builds

The `Nightly Build` workflow (`.github/workflows/nightly.yml`) tags `develop`, runs the Python (`python-tests.yml`), JavaScript (`js-tests.yml`), and Playwright (`playwright.yml`) suites against that tag. It publishes `streamlit-nightly` to PyPI only after every test job passes. Rerunning the failed jobs also reruns the jobs that depend on them, so a successful rerun still publishes that night's build.

The goal is a green nightly with the least work: rerun what was transient, fix what is broken, and hand the rest to a maintainer.

Inputs

A Nightly Build run ID or URL. Without one, use the most recent failed run of `nightly.yml`.

Workflow

- [ ] Collect the failed jobs, steps, and first errors
- [ ] Compare against history
- [ ] Classify: rerun, fix, or report
- [ ] Act on the decision
- [ ] Write the summary

1. Collect the failure

• List the failed jobs and failed steps of the run's latest attempt. `gh run view <id> --attempt <n>` shows earlier attempts.
• Read the failed logs (`gh run view <id> --log-failed`) and find the first real error in each failed job. Ignore cascading failures and skipped jobs.
• Note which variants failed: Python version, browser, test file.
• For Playwright failures, the run uploads a `playwright_test_results_*` artifact with traces and a `snapshot-updates/` folder holding the CI-rendered images. Download artifacts into `work-tmp/`, never into tracked folders.

2. Compare against history

• Check earlier attempts of this run and the last ~10 nightly runs. Nightlies often pass on attempt 2 or 3, so a one-off failure is a good rerun candidate.
• Check recent `push` runs of the failing workflow on `develop`. A failure that repeats there is likely a real breakage.
• Check the commits between the last green nightly tag and this one (`git log <previous-tag>..<this-tag>`). A deterministic failure usually traces back to one of those commits.
• For a test that might be flaky, check for `@pytest.mark.flaky` and the output of `uv run scripts/fetch_flaky_tests.py`.

3. Classify

| Decision | Signals | |----------|---------| | **rerun** (transient) | Network or registry errors (yarn, uv, pip, apt, GitHub API 5xx, `ECONNRESET`), runner lost or shut down, artifact upload or download errors, timeouts during setup. A single test failing on one browser or Python version that passes elsewhere in the run and in other recent runs. Errors that point at the environment instead of the code, such as `Invalid file descriptor` in one pytest worker. | | **fix** (real breakage) | The same failure across browsers, Python versions, or attempts. Lint, type, or build errors. Snapshot mismatches in every browser. The failure reproduces on current `develop`, or started with a specific commit. Tests that depend on the current date and break on a specific day also count, even if they would pass tomorrow. | | **report** (needs a maintainer) | Publishing and release steps: PyPI upload, tag creation, S3 upload. Missing secrets or permissions. An external outage that will not clear soon. The rerun limit is reached and no clear fix exists. `develop` or an open PR already fixes the failure. |

Keep these in mind:

• Playwright retries a failing test once (`--reruns 1` in `playwright.yml`), except failures whose message contains `Missing snapshot` (`--rerun-except` in `e2e_playwright/pytest.ini`). A retried failure in the log failed twice in that attempt. It can still be flaky when it is isolated to one browser and passes in other runs.
• Do not rerun publishing failures. Part of the release may already be on PyPI, and PyPI rejects files that already exist, so a rerun fails the same way.
• If any failure in the run is a real breakage, choose **fix**. A rerun cannot turn the nightly green while one job is broken.
• Allow at most 3 attempts per run (the original plus 2 reruns). After that, a remaining flaky failure needs a fix or a report.
• Before fixing, check whether the failure is already handled: commits on `develop` since the nightly tag that touch the failing area, and open PRs that mention the failing test or error. This includes PRs on `ai-fix-nightly-*` branches from earlier runs and on `ai-fix-develop-ci-*` branches, which fix the same breakage when it first fails on `develop`. If one exists, choose **report** and link it.

4. Act on the decision

**rerun:** Outside CI, rerun only the failed jobs (`gh run rerun <id> --failed`). In CI, record the decision and let the workflow rerun the jobs.

**fix:**

1. Start from the latest `develop`, not the nightly tag. The tag adds a version-bump commit that must not end up in a PR. 2. Reproduce the failure with the matching `make` target or test command. For E2E tests, build the frontend with `make frontend-with-profiler` (as CI does) and run `make run-e2e-test <file>`. 3. Fix the root cause with the smallest change. Do not skip or delete tests, loosen assertions or snapshot thresholds, or add `@pytest.mark.flaky` to turn the build green. For flaky E2E tests, follow the `fixing-flaky-e2e-tests` skill. 4. Update snapshots only when the visual change is intended, such as a commit that deliberately changed the UI. Take Linux snapshots from the run's `snapshot-updates/` artifact instead of rendering them on another OS, and copy only the images for the failing tests. 5. Verify: rerun the failing tests and `make check`. 6. Outside CI, finalize the change and open the PR with the `finalizing-pr` skill. Use a `[fix] ...` title. In the description, link the nightly run and name t