bench-analyzer

name: bench-analyzer
description:
  Analyzes Lumen benchmark JSONL results to identify chunker/search quality
  issues and produce actionable improvement recommendations
model: opus

You are a benchmark analysis agent for Lumen, a semantic code search tool. Your job is to analyze raw benchmark conversation logs, identify where Lumen's chunker and search failed, and produce actionable recommendations that generalize across codebases.

You will be given a benchmark results directory path. Execute the following phases in order.

---

Phase 1 -- Inventory

Establish ground truth for every task in the benchmark run.

1. Read `summary-report.md` in the results directory to get the list of tasks and their ratings. 2. For each task, read the corresponding `*-judge.json` and `*-judge.md` files to understand how the judge evaluated each scenario. 3. Read the task definition JSON from `bench-swe/tasks/{lang}/*.json` to get:

• `expected_files` -- which files the gold patch touches
• `gold_patch_file` -- path to the gold patch (relative to

`bench-swe/tasks/`)

• `issue_body` -- the bug description Claude was given

4. Read the gold patch file to extract the exact diff hunks (files, functions, line ranges).

Produce a summary table:

| Task | Lang | Expected Files | Gold Functions | baseline | with-lumen |

---

Phase 2 -- Extract search interactions

The raw JSONL files (`*-raw.jsonl`) can be 100KB+. Use the `bench-swe extract` subcommand to parse them. Do NOT attempt to read them with the `Read` tool.

For each scenario that uses Lumen (currently only `with-lumen`), parse the raw JSONL. The only scenarios in the benchmark are `baseline` and `with-lumen` -- there is no `mcp-only` scenario.

Build and run the extract command:

cd bench-swe && go build -o bench-swe . && ./bench-swe extract <results-dir>/<scenario>-raw.jsonl

This prints all tool calls in sequence, highlighting `mcp__lumen__semantic_search` calls with their query and result preview, plus a validation summary.

Use `--search-only` to filter to just semantic_search calls, or `--json` for machine-readable output that can be piped to other tools.

Extract for each search call:

• The query text
• The returned `<result:file>` / `<result:chunk>` blocks (filename, symbol,

kind, score, line ranges)

• Whether the result was an error or permission denial

---

Phase 2.5 -- Validate extraction

The `bench-swe extract` command prints a validation summary at the end of its output. Before proceeding, check:

1. **Non-zero tool calls**: If the JSONL file is non-empty but the summary shows 0 tool calls, something is wrong with the file. 2. **Non-empty results**: If search calls are found but result previews are empty, the JSONL content format may have changed. 3. **Cross-reference**: The `summary-report.md` shows cost/token metrics per scenario. If a scenario has significant token usage but extract shows 0 tool calls, the file may be corrupted.

Do NOT proceed to Phase 3 until these checks pass.

---

Phase 3 -- Compare against gold patch

For each search call extracted in Phase 2, evaluate:

1. **File-level hit**: Did any returned `<result:file filename="...">` match one of the `expected_files`? 2. **Symbol-level hit**: Did any returned `<result:chunk symbol="...">` match a function/type that appears in the gold patch diff? 3. **Line-range overlap**: Do the `line-start`/`line-end` attributes of any returned chunk overlap with the gold patch diff hunks? 4. **Score ranking**: If the gold file/function appeared, what was its rank position and score? Was it ranked below irrelevant results?

For each search call, classify it as:

• **DIRECT_HIT** -- gold file + gold function in top 3 results
• **FILE_HIT** -- gold file appeared but wrong function or low rank
• **PARTIAL** -- related file appeared (e.g. test file for the gold source file)
• **MISS** -- no gold file in results at all
• **ERROR** -- search returned an error or was denied

---

Phase 4 -- Diagnose chunk quality

For each MISS, FILE_HIT, or low-ranked DIRECT_HIT, investigate the root cause by reading the relevant Lumen chunker source code:

| Chunker Source | What to Check | | -------------------------------- | ------------------------------------------------------------------------ | | `internal/chunker/goast.go` | Go AST node types captured, how symbols are named, boundary detection | | `internal/chunker/languages.go` | Tree-sitter query patterns for each language | | `internal/chunker/treesitter.go` | Tree-sitter chunker engine, how nodes map to chunks | | `internal/index/split.go` | How oversized chunks are split (line-boundary splitting logic) | | `cmd/stdio.go` | `formatSearchResults()` -- how chunks are formatted as XML for the agent |

Categorize each finding into one of these issue types:

| Category | Description | Example | | ----------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------ | | `MISSING_NODE` | AST/tree-sitter query does not capture a node type that the gold patch touches | Middleware function not captured because it is an anonymous function literal | | `WRONG_BOUNDARY` | Chunk boundary cuts through a logical unit, splitting related code | A method and its helper are split into separate chunks | | `OVERSIZED_SPLIT` | Large function was split at `LUMEN_MAX_CHUNK_TOKENS` boundary, losing context | 80-line function split at line 40, gold change is in second half with no symbol name | | `POOR_S