/swarming-to-green

name: swarming-to-green
description: Runs a long swarm of hundreds to thousands of cheap headless coding workers that drive a large failing-test backlog to green. Failures are clustered by signature, each worker fixes one cluster in its own git worktree, a serial no-regression merge gate lands the fixes, the loop resumes across restarts for hours, and it federates across sibling sessions over git branches when one container's CPU is not enough. Use for "fix these thousands of failing tests with a Haiku swarm", "drive the test suite to green with hundreds of agents", "build this compiler or interpreter against a conformance suite", "launch, relaunch or check in on the swarm loop", "the swarm is stuck, idle or rejecting everything", "parallel agents fixing tests keep hitting merge conflicts and regressions", "add sibling containers to the swarm", or "compare models on the tail the swarm could not fix". Ships the harness, a per-project TOML config, a fake-worker selftest, check-in signals and the 2026-10-12 haiku-cc failure catalogue.
metadata:
  version: 0.7.0

Swarming to green

A loop that turns a test suite's failures into merged fixes, unattended, for hours:

known.json --cluster by signature--> work items --one headless worker each, own worktree-->
  result JSON --serial gate: apply on main, compat check, run smoke + targets + sample,
  zero regressions, >=1 target flips--> fast-forward main --> re-check the whole cluster --> repeat

Every mechanism in it fixed a failure that cost hours on 2026-10-12, when Haiku 5.5 workers built hcc (a C compiler in Python) against ~42K tests: 1,051 hub workers, 464 merges, $24.88 of worker spend by 18:30Z. The catalogue of those failures, each with its signal and where it is handled, is [references/run-2026-10-12.md](references/run-2026-10-12.md).

SK=/mnt/skills/user/swarming-to-green       # or <claude-skills clone>/swarming-to-green
SWARM="python3 $SK/scripts/swarm.py"             # every command below; --root defaults to the git toplevel

When to use it, and when not

Use it when **all** hold: a test command can report every test's status as JSON; there are hundreds or more failing tests; a fix can be checked by re-running tests; and the work will run for hours, longer than anyone watches.

| situation | use instead | |---|---| | fewer than ten tasks, e.g. setup, a suite builder, a refactor | the Agent tool directly (on 2026-10-12, 3 setup agents were run through claude -p for no gain) | | a one-off fan-out of independent tasks (judges, extractors, per-file reviews) with no merging | `dispatching-subagents` | | which model or effort a worker gets, and how to escalate | `agent-routing` (this skill's measured answer for test-fix work is in "Which model" below) | | a fixed multi-step pipeline with no model in the loop | `flowing` | | parallel Claude calls from claude.ai chat | `orchestrating-chat-agents` (`orchestrating-agents` is its deprecated name) | | a handful of failing tests | fix them yourself, or one Agent-tool worker per cluster | | no machine-readable test output yet | write the runner first ("Project setup"), then come back |

**Abandon the run** when the signals say the harness, not the workers, is the problem: `SAME FAILURE`, `DENIALS` or `GATE REJECTS ALL` twice in a row after a fix. Stop the loop (`$SWARM stop`), fix the cause, `unpark` if the fix invalidates earlier attempts, relaunch. Spending through a harness bug is how the abandoned tail was made.

Preflight checklist

Do every item before the first real launch. Each one cost hours when missed.

1. **Failing-test supply first.** Haiku drained haiku-cc's conformance suites in 70 minutes and the queue sat empty. Have thousands of failing tests, a tier of real programs (one build failure blocks hundreds of tests), and a suite builder that works in its own worktree and commits to main. The loop baselines new suites within two minutes. Details: [references/project-setup.md](references/project-setup.md). 2. **Plan containers now if workers do real compute.** When each worker compiles and runs tests, CPU caps the swarm whatever the launch mechanism: 4 vCPU held haiku-cc at load 10-22 for eight hours before siblings were added. Decide the partition and the number of sibling sessions at preflight: [references/scaling-out.md](references/scaling-out.md). 3. **Disk.** `df -h /tmp <root>/..` needs 10 GB or more free. Worker scratch reached 14 GB and stale worktrees 5.6 GB on 2026-10-12. The loop deletes scratch on reap, sweeps stale worktrees at start, and stops launching below `run.min_free_gb`. 4. **The 2-hour background cap.** A background Bash job dies at 2 hours, and at 30 minutes if you omit `timeout`. Launch with `timeout: 7200000` and `--max-minutes 110`; the loop exits cleanly, workers keep running, and the next launch adopts them. 5. **Coordinator models.** Every session you create inherits your model unless you pass `model`. Idle Opus siblings cost about $1.40 per check-in. Give siblings and experiment sessions a Sonnet or Haiku model. 6. **Push namespace.** Federation pushes `<prefix>main/state/cand/*`. A push-policy guard (claude-workspace `scripts/guards.py` `_ALLOWED_PREFIXES`) must allow the prefix. 7. **Gitignore build artefacts and pin the language version** the fast interpreter supports; set `gate.compat_command` to enforce the pin.

Setup

1. **Config.** Copy [assets/swarm.example.toml](assets/swarm.example.toml) to `<root>/swarm.toml` and fill `tests.command`, `paths.code/allowed/forbidden`, `queue.ladder`, `gate.always_suites/cheap_suites`, `gate.compat_command`, `worker.system_prompt`, `worker.repro_command`, and `worker.deny` for whatever the workers must never run (network fetches when the answer is on the internet). Every key: [references/config.md](references/config.md). A complete working config for haiku-cc is [assets/haiku-cc.toml](assets/haiku-cc.toml). 2. **W