multi-agents

multi-agents

Multi-agent teams for real work: measured, not assumed.

Docs · Quickstart · Teams · Benchmark · Roadmap

Featured in Tech Forum: Does a Team of AI Coding Agents Beat One?

North star: most agent orchestrators skip one question, and the benchmark here answers it: did the team beat a single agent on the same task, and what did the extra agents cost?

Run a team of coding agents on a task in your repository: a single agent, a coder ⇄ reviewer loop, or several agents with a judge. Workers are the coding CLIs you already use (Claude Code, Codex) or a built-in agent that can use any model through the OpenAI Agents SDK. Each worker gets its own git worktree. You get one patch back, plus an itemized record of every agent call: tokens, cost and seconds. Your working tree is untouched unless you pass --apply.

Why measure

The evidence on teams of coding agents points both ways:

• A 260-configuration study found multi-agent setups 2–15% worse than single agents on SWE-bench Verified (Towards a Science of Scaling Agent Systems).
• A team with a coordinator, a researcher, an engineer and a reviewer reached 72.2% on SWE-bench Verified, 7.2 points better than mini-SWE-agent but only 0.4 above the strongest single-agent baseline, with different models per role (Agyn). A decentralized pair, a patch editor and a validator with isolated context, gained 3.6 and 8.4 points over mini-SWE-agent (icat-agent).
• The best-known coding benchmark is contaminated: OpenAI stopped reporting SWE-bench Verified in February 2026.

So the answer depends on the topology, the models and the task, and published numbers may be contaminated. This project measures it on tasks created after the models' training cutoffs, with paired comparisons and cost.

Quickstart

Requirements: Python 3.10+, uv, git, and at least one logged-in coding CLI (claude or codex). CLI harnesses use your existing login; no API keys needed. The package is multi-agent on PyPI and the command is multi-agents: run it with uvx as below, or keep it installed with uv tool install multi-agent (or pipx install multi-agent).

uvx --from multi-agent multi-agents run \
  --repo ~/code/myproject \
  --team 'best:workers=claude@haiku+codex@gpt-6-luna' \
  "Make the tests in test_slug.py pass without editing the tests."

Real output from that command on a small repository:

Team: best:workers=claude@haiku+codex@gpt-6-luna,judge=claude@haiku
role   ┃ harness@model           ┃ tokens in/cached/out ┃    cost ┃ secs
━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━
worker │ claude@claude-haiku-5-5 │       236k/201k/1.5k │ $0.0096 │ 34.8
judge  │ claude@claude-haiku-5-5 │       194k/157k/2.6k │ $0.0102 │ 42.8
worker │ codex@gpt-6-luna        │         107k/94k/667 │ $0.0025 │ 31.6
judge  │ claude@claude-haiku-5-5 │       192k/149k/3.8k │ $0.0119 │ 52.3
Total: 729k/604k/8.5k tokens, cost $0.0342, 85.0s
scores: [8.0, 8.0]
chosen: 1
Files changed: 1
  slug.py  +6 -3

The run prints the exact git apply command for the final patch and saves a JSON record outside your repository (~/.cache/multi-agents/runs/).

Teams

H is claude, codex or sdk, optionally with @model, for example claude@haiku, codex@gpt-6-luna or sdk@litellm/openrouter/<model>. check (the --check command) can be the reviewer or the judge: a test-driven loop with no LLM reviewer. Details: team runner guide.

From issue to pull request

--check runs your tests (or build, or lint) against the final patch in a separate worktree and records pass/fail; with an LLM judge, candidates that fail it rank below those that pass. --issue takes the task from a GitHub issue, and --pr commits the result to a new branch and opens a draft pull request with the run summary, without touching your checkout.

multi-agents run --repo . --issue 42 --pr \
  --team 'plan:planner=claude@haiku,workers=claude@haiku+codex@gpt-6-luna' \
  --check 'python3 -m unittest -q'

A real plan run on a repository with two unrelated bugs (without --issue/--pr):

role    ┃ harness@model           ┃ tokens in/cached/out ┃    cost ┃ secs
planner │ claude@claude-haiku-5-5 │       187k/150k/2.0k │ $0.0101 │ 16.8
worker  │ claude@claude-haiku-5-5 │        141k/104k/510 │ $0.0088 │  8.7
worker  │ codex@gpt-6-luna        │          55k/37k/349 │ $0.0024 │  9.7
Total: 385k/291k/2.9k tokens, cost $0.0212, 26.8s
subtask 1 [applied]: Fix word_count in textstats.py (claude@haiku)
subtask 2 [applied]: Fix format_cents in money.py (codex@gpt-6-luna)
check: passed (exit 0, 0.0s) python3 -m unittest -q

Issue text is untrusted input to the agents: don't combine --issue on public repositories with --permissions bypass or the sdk harness.

Use it from your coding agent

This repository is also a plugin marketplace for both coding CLIs, so a running agent can hand a task to a team and then show you the patch. Install it once:

# Claude Code
claude plugin marketplace add BichengWang/multi-agents
claude plugin install multi-agents@multi-agents

# Codex
codex plugin marketplace add BichengWang/multi-agents
codex plugin add multi-agents@multi-agents

Then, inside Claude Code (the optional spec comes first; the default is a cheap coder ⇄ reviewer pair that pairs the other CLI with the one you are in):

/multi-agents:team make the tests pass
/multi-agents:team best:workers=claude@haiku+codex@gpt-6-luna fix the flaky retry in client.py

In Codex, ask for the skill by name, for example Use $multi-agents:multi-agents to make the tests pass. Either way the agent runs the published CLI, shows you the summary and the patch, and applies it only after you say so. The skill is in plugins/multi-agents.

Safety and cost

• Permissions. By default (--permissions safe) each CLI runs inside its own sandbox: edits and test commands are confined to the worktree. --permissions bypass turns those checks off and is meant for disposable machines and containers only. The sdk harness has no permission system.
• Cost. Costs come from the CLI's own report or from list prices with a cited source. A run that cannot be priced shows as unpriced and makes the total unpriced; nothing is guessed.
• Budget. --max-cost USD stops new agent runs once that much has been spent; the team then returns the patch it has so far. It is a stop on new work, not a ceiling: runs already in flight finish (parallel workers run side by side), so the total can overshoot by their cost. An unpriced run cannot be counted, so after one the budget refuses further runs. The CLI therefore rejects a team with an unpriced harness before it starts, unless you pass --allow-unpriced, which counts those runs as $0 and leaves their spend outside the limit. The record and summary show the spend and any refused runs.

Benchmark

The benchmark runs team specs against solo agents on GitHub issues opened after the models' training cutoffs (SWE-bench-Live MultiLang: Rust, Java, Go, C/C++, C#, JS/TS), each in a fresh sandbox (Modal, or --backend docker on your own Docker daemon) with git history stripped. Patches are graded with the dataset's official hidden tests. The report gives resolve rate with 95% confidence intervals, paired wins and losses against the solo baseline on the same tasks (exact McNemar test), cost per task and per resolved task, and wall time. Timeouts are graded rather than dropped, so slow teams cannot shrink the comparison to the easy tasks.

uv sync --locked --extra bench
python -m agent.bench tasks --since 2026-07-01 --limit 5 --seed 7    # no credentials needed

Setup, methodology, cost estimates and limitations: benchmark guide.

First results (October 2026): on 48 fresh tasks, Claude Haiku 5.5 resolved 27 solo, 29 with a reviewer and 28 as best of 3, at $0.061, $0.098 and $0.17 per task. The teams' one or two extra tasks are not significant (paired p = 0.5 and 1.0): the reviewer approved 47 of 48 final patches, and the judge's top scores were tied on 32 of 48 tasks. Table, analysis and data: pilot results.

Status

Plans are in docs/ROADMAP.md.

Development