/knowledge-extraction

name: knowledge-extraction
description: Sweep chunks for entries that match the project's schema and emit them via the event log. Use whenever the chunk phase has produced chunks and the extract phase is next, when the user says "extract X from the corpus," or when [[ralph-loop]] advances into the extraction phase. Subagent fan-out is the norm; each subagent processes one chunk; canonical state lives in the reducer's output.
metadata:
  triggers:
    - extract knowledge
    - extract entries
    - sweep the chunks
    - fan out extraction
    - run the extractor
    - extract rules
    - extract facts
    - extraction phase

knowledge-extraction

The phase where chunks become entries. This is where the vertical axis of John's matrix earns its keep — hundreds of subagents in parallel, each processing one chunk, each emitting events the reducer folds into canonical state. Without subagent fan-out, this phase doesn't scale.

Where the work happens

• **Inputs**: `<project>/.john/chunks/<chunk-id>.md` + `<project>/.john/chunks/chunks_index.json` (from [[chunking]])
• **Schema reference**: PLAN.md app-type definition section (per [[schema-design]])
• **Outputs**: subagents emit to `<project>/.john/events/extract/<chunk-id>/<subagent-id>-*.json` (one file per event; exact event shapes and filename suffixes are in the `knowledge-extractor` agent definition); reducer (`${CLAUDE_PLUGIN_ROOT}/scripts/reduce_events.py extract`) folds to `<project>/.john/checkpoints/extract/state.json`; canonical state then drives [[knowledge-rewrite]].

The MECE sweep

Extract "everything there is" OR "everything needed for what" — which one depends on the project's intent. Decide that early and let it shape the sweep.

• **Comprehensive sweep**: "extract everything there is in this corpus that matches the schema." Right for encyclopedic projects, regulations, broad knowledge bases.
• **Goal-directed sweep**: "extract everything needed to answer X." Right for narrow apps where coverage outside the goal is wasteful.

Either way, MECE applies to coverage within the chosen scope: don't extract the same entry twice; don't leave the scope partially covered. Dedup across chunks happens later, in the rewrite phase (see [[knowledge-rewrite]]'s two-tier dedup) — the shipped reducer folds events without deduplicating; your job is to give that pipeline good raw events.

Fan-out per chunk

For each chunk, dispatch a subagent. Brief them comprehensively (per [[subagent-dispatch]]'s checklist):

1. **Project intent** (from PLAN.md top). 2. **The chunk** they're processing — the chunk file path or contents. 3. **The schema reference** — what an entry looks like, what fields, what cross-link semantics. 4. **The event log target** — where to write events (`<project>/.john/events/extract/<chunk-id>/`). 5. **What to return** — a one-line digest. No raw extracted content in the digest; that's in the event log. 6. **What NOT to do** — don't write canonical state directly; don't ask the user; don't try to dedupe across chunks (the reducer does that).

For small corpora (<10 chunks), inline extraction in the main agent context is fine. For real-world corpora (10s-1000s of chunks), always fan out.

The self-correction echo (mathlab pattern)

Borrowed from mathlab's "ops[0] echoes the problem" trick: have each extraction subagent's first action be to **echo back its understanding of the chunk** before extracting from it. This catches misreading, character encoding bugs, and chunks-handed-to-the-wrong-subagent failures cheaply.

Mechanically: the briefing includes the instruction *"Before extracting any entries, emit an event of type `chunk_echo` with a 2-3 sentence summary of what this chunk says. Then proceed."* The reducer folds the echoes into the checkpoint; its completeness check splits severity — a chunk missing `chunk_complete` lands in `incomplete_chunks` (possibly-unfinished work, worth a look before advancing), while a chunk that only skipped its echo lands in `chunks_missing_echo` (an INFO/audit note, *not* a reason to re-extract). YOU spot-check the echoes there — a wildly off-base echo flags a chunk for re-extraction.

Cost: one event per chunk's worth of summarization. Cheap compared to re-running an extraction that silently extracted from the wrong chunk.

Schema iteration during extraction

[[schema-design]] says the schema will iterate. Extraction is one of the phases where iteration surfaces:

• An extractor reports *"this chunk has structure the schema doesn't represent."* The subagent emits a `schema_observation` event to the same event log; these fold into the extract phase's canonical state alongside the other events (filter on `event_type` when reviewing; a template's custom reducer may split them into a separate observations array).
• You (the main agent, per [[ralph-loop]]) review these observations after the phase-fanout completes. If N≥3 observations point at the same gap, surface a schema-extension question to PLAN.md's Open Decisions and ask the user before proceeding.
• [[knowledge-rewrite]] also reads these observations during the rewrite phase to guide cross-linking and dedup decisions — observations may flag entries that should be kept separate despite similarity.
• Don't re-extract the entire corpus on every schema change. Use corrective events instead: a new `entry_replaced` event supersedes the older one. The reducer's fold function handles supersession deterministically.

Model tier per chunk

John core uses the active runtime's model routing. For extraction subagents:

• **Sonnet** is usually right. Extraction needs decent reading comprehension + schema adherence; Sonnet has both.
• **Haiku** works for highly structured chunks (e.g., extracting facts from a clean table). Cheap, fast.
• **Opus** for chunks with judgment-heavy content (e.g., extracting *intent* from a legal preamble where the wording is deliberately vague).
• **WorkerLLMs** (SiliconFlow, DeepSeek, etc.) where