/octocode-benchmark

name: octocode-benchmark
description: "Use when planning, running, grading, or reporting the by-hand Octocode research benchmark — pairwise matchups (Octocode anchor vs one baseline: gh+RTK, gh+Headroom, or plain gh) over markdown questions, with a fresh isolated runner agent per (question, arm, pass), one blind judge per question grading two answers X/Y in randomized order, and an orchestrator that summarizes accuracy/quality/workflow/characters. Results measured in total characters through the model (model-in delivered + model-out commands/args + final answer)."

Octocode benchmark

Plain-markdown, run-by-hand CLI research comparison. Octocode is the **anchor**; each baseline is a **separate pairwise matchup** (`octocode` vs `rtk` | `headroom` | `gh`). Per question, per pass: two isolated runners answer, one blind judge grades them **X / Y** (randomized per question). Run ≥3 passes; the rollup shows every matchup together. No harness, no JSON.

Paths below are relative to the package root `packages/octocode-benchmark/`. Shared tooling lives in `compare/bin/`; questions in `compare/github-questions/`; reports in `results/`.

Flow (4 phases)

0. **Preflight** — verify + pin every arm; a failure invalidates the run. 1. **Answer** — 2 isolated runners (anchor + baseline) per question/pass, leanest-legal path, each appends a `## Q<n>` section to `answers/<arm>-p<pass>.md`. 2. **Judge** — after both sections exist, one blind judge reasons to a verdict, then scores. 3. **Summarize** — validate logs, aggregate paired stats, update the rollup.

How it is MEASURED (characters, never self-reported)

The metric is **`total_chars = model-in + model-out`** in Unicode code points, from an instrumented log — the *tool transcript only* (excludes system prompt, tool schemas, model reasoning; the fixed per-arm primer is excluded by rule; any later help/schema/failed call **is** counted).

• **model-out** = the command string + args the model wrote, plus the final answer.
• **model-in** = the tool output pulled back into context (for Headroom, the *compressed* output).

Mechanism: every research command runs through its arm's thin wrapper, which shells the real CLI unchanged, prints output verbatim, and appends one JSONL row per call:

| Arm | Wrapper | Runs | Log env | |---|---|---|---| | octocode (local build) | `compare/bin/octoc` | `npx octocode tools …` | `OCTO_LOG` | | octocode (published pin) | `compare/bin/octoc1822` | `npx -y octocode@18.2.2 tools …` | `OCTO_LOG` | | gh+RTK | `compare/bin/rtkm` | `rtk gh …` | `RTK_LOG` | | gh+Headroom | `compare/bin/ghc` | `gh …` → Headroom compress | `GHC_LOG` (+ `HR_PY`) | | plain gh | `compare/bin/ghm` | `gh …` (read-only) | — |

The final answer is logged as pure model-out via `compare/bin/record_answer.py`. Per-question total = `compare/bin/sumlog.py --strict <log>`; the whole campaign is checked byte-faithfully by `compare/bin/validate_campaign.py`. **Never trust a hand-counted number** — recompute from the JSONL. Only `elapsed_ms` (octocode/rtk/gh) is captured for time; it is not a fair latency metric (npx bootstrap per call, no Headroom timing) — do not headline it.

How it is SCORED (blind judge, correctness-first)

One blind judge per question grades the two answers as **X / Y** (order randomized per question, tool identity redacted). It **reasons to ground truth first**, then scores each answer: **correctness 0–10, research depth 1–5, workflow 1–5** (rubric in `references/JUDGING.md`).

Decision per pairing: if one arm is net strictly more correct (paired sign test) it **wins** — a confidently-wrong answer never wins on footprint. If correctness is statistically tied, **characters decide** by the **geometric-mean of per-question ratios** (baseline ÷ octocode) + median + leaner win-rate + bootstrap CI — never a pooled sum alone. Aggregate paired, per question, over **≥3 passes**. Method + worked example: `references/aggregation-and-stats.md`.

Quickstart (copy-paste, one matchup, one pass)

cd packages/octocode-benchmark
export HR_PY="$HOME/.local/share/uv/tools/headroom-ai/bin/python"   # Headroom arm only

# Phase 0 — preflight (non-zero exit = fix before running)
bash skills/octocode-benchmark/scripts/check-prereqs.sh 18.2.2

# set up a campaign dir
CAMP="campaigns/run-$(date -u +%H%M%S)-$(date -u +%Y-%m-%d)"; mkdir -p "$CAMP/answers" "$CAMP/judge"

# Phase 1 — answer (spawn ONE isolated agent per arm; never mix arms in an agent).
# Each research call sets its per-question log, e.g. octocode Q4:
OCTO_LOG="$CAMP/octocode-p1-Q4.jsonl" ./compare/bin/octoc1822 ghGetFileContent \
  --queries '{"owner":"axios","repo":"axios","path":"lib/adapters/http.js","matchString":"follow-redirects"}'
# baseline (gh+RTK) Q4:
RTK_LOG="$CAMP/rtk-p1-Q4.jsonl" ./compare/bin/rtkm search code --repo axios/axios follow-redirects --limit 20
# log the final answer, then append a "## Q4" section (Answer + Research steps) to answers/<arm>-p1.md
python3 compare/bin/record_answer.py --log "$CAMP/octocode-p1-Q4.jsonl" --question Q4 --file answer.txt

# Phase 2 — judge: build the blind packet, then one reasoning-first verdict per question
python3 compare/bin/build_blind_packet.py --help          # X/Y randomized, tool identity redacted

# Phase 3 — validate + aggregate + report
python3 compare/bin/sumlog.py --strict "$CAMP/octocode-p1-Q4.jsonl"
python3 compare/bin/validate_campaign.py "$CAMP" --question-count 30
python3 compare/bin/per_question_summary.py --out results/PER_QUESTION_SUMMARY.md --json results/per_question_summary.json

Runner/judge briefing packets, spawn scaling (batch Q1-15/Q16-30 within one arm), and output layout: `references/run-with-agents.md` → `run-preflight.md` + `run-phases.md`.

Hard gates (skip one and the run is worthless)

• **Isolation** — a fresh agent per (question, arm, pass) + a separate judge; no shared transcript, no answer key. Batch questions within an arm, never mix arms.
• **Fairness** — leanest-legal path o