/benchmark

name: benchmark
description: "Run SWE-bench Lite benchmarks against agtx coding agent workflows. Guides setup, configuration, execution, evaluation, and reporting."
disable-model-invocation: true

Benchmark — SWE-bench Lite

You are a benchmark guide. Help the user run, configure, and evaluate SWE-bench Lite benchmarks against agtx agent workflows.

**Answer questions, surface the right commands, and walk through setup interactively.** All commands assume the user is in the `benchmark/` directory.

---

Prerequisites

| Tool | Install | |------|---------| | **Docker** | Required for sandbox mode. macOS: Docker Desktop. Ubuntu: `apt install docker.io` | | **agtx binary** | `cargo build --release` from repo root | | **uv** | `curl -LsSf https://astral.sh/uv/install.sh \| sh` | | **tmux** | macOS: `brew install tmux`. Ubuntu: `apt install tmux` | | **tokscale** (optional) | `npm install -g tokscale` — enables cost/token tracking in results | | **Coding agent** | At least one: Claude Code, Gemini CLI, or Codex CLI |

One-Time Setup

cd benchmark/swebench

# Initialize Python environment (once, or after pyproject.toml changes)
uv sync

# [Sandbox only] Build the tools image (tmux + Node.js + Claude Code)
python prebake_images.py --verbose

# [Sandbox only] Build the Linux agtx binary (Ubuntu 22.04 / glibc 2.35)
bash build_linux_binary.sh

The tools image populates the shared Docker volume `agtx-swebench-tools` on the first benchmark run. To force a refresh after updating Claude Code:

docker volume rm agtx-swebench-tools
python prebake_images.py --force --verbose

---

Configuration

Config files live in `swebench/configs/`. Each is a standard agtx `ProjectConfig` TOML written to `.agtx/config.toml` in every cloned repo.

**Minimal (no workflow):**

default_agent = "claude"
workflow_plugin = "void"

**Standard agtx workflow:**

default_agent = "claude"
workflow_plugin = "agtx"
worktree_dir = ".agtx/worktrees"

**Sandbox-optimised** (agent works directly in `/testbed`, no worktree):

default_agent = "claude"
workflow_plugin = "agtx"
worktree_dir = ".agtx/worktrees"
skip_worktree = true

**Mixed agents** (different agent per phase):

default_agent = "claude"
workflow_plugin = "agtx"

[agents]
planning = "gemini"
running  = "claude"
review   = "codex"

**With `sandbox_init`** (install extra tooling inside the container before the agent starts):

default_agent = "claude"
workflow_plugin = "agtx"
skip_worktree = true

sandbox_init = [
    "curl -fsSL https://raw.githubusercontent.com/rtk-ai/rtk/refs/heads/master/install.sh | sh",
    "export PATH=$HOME/.local/bin:$PATH && rtk init -g",
]

`sandbox_init` commands run with `HOME=/home/bench` and `PATH` including `/home/bench/.local/bin`. Each config activates only the tools it explicitly installs — other configs are unaffected.

Available plugins: `void`, `agtx`, `agtx-terse`, `gsd`, `spec-kit`, `bmad`, `openspec`, `superpowers`, `agent-skills`

Pre-built configs for common combinations are in `swebench/configs/`.

---

Running

> **Sandbox mode requires a Linux x86_64 binary.** Use `--agtx ../target/agtx-linux-x86_64` (not `../target/release/agtx`). > > **Always recommend sandbox mode.** SWE-bench repos require specific Python versions and C extensions that aren't available on the host — agents typically fail to run `pytest` outside the containers. In sandbox mode each task runs inside its official SWE-bench Docker image with the repo pre-installed in a working conda env (`testbed`).

**Single random task:**

uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-void.toml \
  --instances 1 --verbose \
  --agtx ../target/release/agtx

**Specific instance IDs:**

uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-void.toml \
  --instance-ids sympy__sympy-20590 django__django-11099 \
  --agtx ../target/release/agtx

**Sandbox mode (recommended):**

python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --instance-ids astropy__astropy-12907 \
  --sandbox --verbose \
  --agtx ../target/agtx-linux-x86_64

**Parallel tasks:**

uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --concurrency 4 \
  --agtx ../target/release/agtx

**Full 300-task run:**

uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --agtx ../target/release/agtx

**Resume an interrupted run** (pass the same `--output-dir`):

uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --output-dir swebench_output/agtx_claude_20260427_120000 \
  --agtx ../target/release/agtx

**Hard mode** (prose only — no code blocks or stack traces in the problem statement):

uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --hard \
  --agtx ../target/release/agtx

All CLI Options

| Flag | Default | Description | |------|---------|-------------| | `--config PATH` | *(required)* | agtx config.toml for this run | | `--instances N` | all 300 | Run first N tasks | | `--instance-ids ID...` | — | Run specific instance IDs | | `--concurrency N` | 1 | Parallel tasks | | `--sandbox` | off | Run inside SWE-bench Docker images (recommended) | | `--output-dir PATH` | `./swebench_output/{config-name}_{ts}/` | Output directory | | `--workdir PATH` | `/tmp/swebench_repos` | Repo clone directory (non-sandbox only) | | `--agtx PATH` | `./target/release/agtx` | agtx binary (must be Linux x86_64 for sandbox) | | `--phase-timeout SECS` | 1200 | Per-phase max seconds (20 min) | | `--model-name STRING` | `{config-stem}` | Label in predictions.jsonl | | `--split STRING` | `test` | HuggingFace dataset split | | `-