/doing-a-simple-two-stage-fanout

name: doing-a-simple-two-stage-fanout
description: Use when analyzing a large corpus of text, code, or data that exceeds a single agent's effective context - orchestrates parallel Worker subagents, Critic review subagents, and a final Summarizer subagent with task tracking and failure recovery
user-invocable: true

Two-Stage Fan-Out Analysis

Divide a corpus across Worker subagents, review with Critic subagents, synthesize with a Summarizer. Every stage writes to files; every subagent gets its own task.

**Do not use nested subagents.** This workflow may dispatch first-level Worker, Critic, and Summarizer subagents. Those subagents must read their assigned inputs, write their outputs, and return directly to the caller. They must not dispatch additional subagents.

Overview

Corpus → [Workers] → [Critics] → Summarizer → Report

**Workers** each analyze a slice of the corpus. **Critics** each review all Worker reports for a subset of segments, checking for gaps and inconsistencies. A single **Summarizer** reads all Critic reports and produces the final output.

Step 0: Gather Inputs

If the user's intent is not already clear, ask two questions using AskUserQuestion:

**Question 1: What to analyze.** Ask what corpus to analyze and what the analysis goal is. Skip if obvious from context.

**Question 2: Effort level.** Present these options in this order (do not reorder to put recommended first):

| Level | SEGMENTS_PER | REVIEWS_PER | When to use | |-------|-------------|-------------|-------------| | Some effort | 3 | 2 | Default for most analyses | | A lot of effort | 3 | 3 | When thoroughness matters more than speed | | Herculean effort | 2 | 3 | When you cannot afford to miss anything |

Recommend one if you have enough context, by appending "(Recommended)" to that option's label. But keep the options in the order shown above regardless.

**Definitions:**

• `SEGMENTS_PER` — how many corpus segments each Worker processes
• `REVIEWS_PER` — how many independent Critic reviews each segment receives

Step 1: Compute the Layout

You need to determine how many segments, workers, and critics the analysis requires. This depends on corpus size and agent context capacity.

Estimating Corpus Size

If you have file paths, estimate tokens:

• **Prose**: 1 token per 4 characters
• **Source code**: 1 token per 3 characters
• **By word count**: 1 word is roughly 1.33 tokens

Use the Bash tool to count characters: `wc -c file1 file2 ...` or `find /path -type f -exec cat {} + | wc -c`.

For more precise estimates, run the [compute_layout.py](./compute_layout.py) script bundled with this skill:

python3 /path/to/compute_layout.py --corpus-chars 800000 --segments-per 3 --reviews-per 2
python3 /path/to/compute_layout.py --corpus-files file1.txt file2.txt --segments-per 3 --reviews-per 2
python3 /path/to/compute_layout.py --corpus-tokens 200000 --segments-per 3 --reviews-per 2 --json

Computing Manually

If you cannot run the script, compute by hand. **Use the Bash tool with `python3 -c "..."` for all arithmetic** — do not compute in your head.

**Agent capacity:**

AGENT_CONTEXT  = 200,000 tokens
RESERVED       = 35%  (for prompt, reasoning, output)
AVAILABLE      = AGENT_CONTEXT * 0.65 = 130,000 tokens
SEGMENT_BUDGET = AVAILABLE / SEGMENTS_PER

**Segment count:**

OVERLAP  = 10% of SEGMENT_BUDGET
STRIDE   = SEGMENT_BUDGET - OVERLAP
SEGMENT_COUNT = ceil((CORPUS_TOKENS - SEGMENT_BUDGET) / STRIDE) + 1

If `CORPUS_TOKENS <= SEGMENT_BUDGET`, then `SEGMENT_COUNT = 1` (no fan-out needed).

**Agent counts:**

WORKER_COUNT = ceil(SEGMENT_COUNT / SEGMENTS_PER)
TOTAL_CRITIC_ASSIGNMENTS = SEGMENT_COUNT * REVIEWS_PER
CRITIC_COUNT = ceil(TOTAL_CRITIC_ASSIGNMENTS / SEGMENTS_PER)

What These Numbers Mean

• Each **Worker** reads `SEGMENTS_PER` consecutive segments of raw corpus and writes an analysis report.
• Each **Critic** reads all Worker reports that cover a subset of segments and writes a review.
• Each segment gets reviewed by `REVIEWS_PER` different Critics (redundancy for thoroughness).

Assigning Critics to Segments

The critic count tells you how many critics to create, but you also need to decide which segments each critic reviews. Use round-robin assignment to distribute `REVIEWS_PER` critic passes evenly across segments:

For each segment S (1 to SEGMENT_COUNT):
    Assign REVIEWS_PER different critics to review S
    Rotate through critics: critic index = (S * review_pass + offset) % CRITIC_COUNT

In practice, use `python3 -c "..."` to generate the assignment table. Example for 6 segments, 4 critics, REVIEWS_PER=2:

C01 reviews: S01, S03, S05
C02 reviews: S02, S04, S06
C03 reviews: S01, S04, S06
C04 reviews: S02, S03, S05

Each segment appears in exactly 2 critics' lists. Each critic reads the Worker reports that cover its assigned segments. Include this assignment table in the orchestration plan so the mapping is explicit and verifiable.

Step 2: Set Up the Temp Directory

If the user specified a working directory, use it. Otherwise, create one:

WORK_DIR=$(mktemp -d -t fanout-XXXXXX)
mkdir -p "$WORK_DIR/segments" "$WORK_DIR/workers" "$WORK_DIR/critics"

All paths in prompts and file references are **absolute paths**. Subagents cannot resolve relative paths reliably.

Step 3: Enter Plan Mode and Write the Orchestration Plan

Enter plan mode. Write a plan document that includes:

1. **Layout summary**: corpus size, segment count, worker count, critic count, effort level 2. **Fan-out diagram**: a Mermaid diagram showing the pipeline (see [diagram-templates.md](./diagram-templates.md) for syntax). For large layouts (>10 workers), collapse worker ranges (e.g., `W01-W10`) into summary nodes. If the user requests Graphviz instead, use the DOT template from the same file. 3. **Worker assignment table**: which segments each Worker handles (e.g., `W01: S01-S03`) 4. **Critic assignment table**: which segments e