/wiki-ingest

Ingest a source document into the LLM Wiki.

Usage: `/wiki-ingest <file|folder|inbox>` — required argument

**If `$ARGUMENTS` is empty**: print the usage below and **stop**.

Usage: /wiki-ingest <file|folder path | inbox>

Examples:
  /wiki-ingest raw/articles/my-article.md       # single md
  /wiki-ingest raw/NewsScrap                    # HTML article folder
  /wiki-ingest raw/PDF                          # PDF folder
  /wiki-ingest inbox                            # process the mobile share-sheet URL queue

Traversal Pattern

L2-1 source + L2-2 stub cycle + Meta (full rebuild) — Writing-driven, tightly coupled. The Reporter performs the raw → structured conversion end to end.

| Phase | Cycle | Owner | |---|---|---| | Prefilter dedup | (deterministic) | Editor-in-Chief → `tools/_ingest/prefilter_ingest.py` or `_source_map.json` matching | | Pre-fanout wiki read | GROUND common | Editor-in-Chief (Read `index.md`·`overview.md` — shared sub-agent context) | | Source page authoring | L2-1 GROUND·APPLY | Reporter (mode=apply; read body + claim atomization + citation type prefix + grade) | | Stub creation/update | L2-2 GROUND·APPLY | Reporter (mode=apply; auto-generate entity·concept stubs — body ≥200 chars, 2 H2 required) | | Person incumbency check | GROUND external | Reporter (mode=ground, WebSearch) | | Source verification (1st) | VERIFY₁ | Reporter self (right after authoring, confirm `python tools/lint.py source <slug>` PASS — self-ADAPT on FAIL) | | Source verification (2nd, sub-trigger) | VERIFY₂ | Editor-in-Chief — invoke the Desk on sources meeting `[fact]≥7 AND citations≥3` (based on the `tools/_lint/source.py` Advisory auto-surface, batched at the post-fanout stage). SoT: [`agents/desk.md`](../agents/desk.md) L2-1 source VERIFY₂. | | Hub verification (1st) | VERIFY | Reporter (single-source stub during fanout) · Editor-in-Chief (post-fanout cross-source stub) — `python tools/lint.py hub schema` | | Cascade update | (deterministic) | Editor-in-Chief → `tools/build.py` (graph → clusters → contradictions → index → dependencies) + hub `sources:` sync | | Batch verification (full) | VERIFY | Editor-in-Chief — post-fanout `python tools/build.py` + `python tools/lint.py` all groups (deterministic check, so done directly by main — the deliberate lint cycle that needs `lint-report.md` output is delegated to the Copy Editor = `/wiki-lint`) | | Log append + defect logging | — | Editor-in-Chief (`log.md` append + batch this cycle's lint FAILs and Desk-actionable defects → `tools/log_defect.py` corpus). SoT: [`agents/editor-in-chief.md`](../agents/editor-in-chief.md) self-evolution automatic channel |

Inbox Mode (`/wiki-ingest inbox`)

A shortcut path to fetch + ingest the URL queue accumulated in `raw/_inbox.md`. Because the mobile share-sheet, `/wiki-news` interactive, and `--gap` channels all converge on this queue, the 2-stage fetch (deterministic + WebFetch fallback) is owned solely by this mode.

1. **1st-stage deterministic fetch** — `python tools/_ingest/fetch_inbox.py`. Reads `raw/_inbox.md` line by line and (a) matches against `_source_map.json::by_url` → SKIPPED + dequeue, (b) for new ones, fetches via the `tools/_ingest/fetch_article.py` logic → `raw/NewsScrap/<slug>.md` or `raw/PDF/<slug>.pdf`, (c) failures stay in inbox + a `FAILED:<reason>` record in today's section of `raw/_archive.md`. 2. **2nd-stage WebFetch fallback** — for URLs the 1st stage left behind as `FAILED:short-content` (JS-rendered sites), the Reporter (mode=ground) fetches the body via `WebFetch` and writes `raw/NewsScrap/<slug>.md` directly. The frontmatter (`source` (URL)·`created`·`description`, etc.) must be **character-for-character identical** to the 1st-stage tool output — arbitrary changes break `_source_map.json` matching. If the body is under 100 chars, hold off on ingest. Successful URLs are removed from `_inbox.md`. (`BLOCKED`·`HTTP-4xx`·network failures are not fallback targets — they stay for retry.) 3. **Proceed in folder mode** — the 12-step procedure on the fetch outputs (`raw/NewsScrap`·`raw/PDF`). **Create source pages only after the raw has landed on disk** — because the raw tree is the dedup ground truth (`by_path`), an unfetched source bypasses duplicate detection. 4. **Pre-approved commit of system-managed files**: `raw/_inbox.md`·`raw/_archive.md` are system-managed files, so commit·push them automatically without separate approval. Other raw content files and ingest outputs (`wiki/`·`graph/`) are committed only after explicit approval from the wiki operator.

Setup guide: [`.claude/operations/mobile-inbox-setup.md`](../operations/mobile-inbox-setup.md)

Prefilter (Folder Mode)

• **Automation first** — `python tools/_ingest/prefilter_ingest.py <raw-folder>`. by_url first + by_path second fallback + Raindrop exclusion + 0-byte skip, deterministic classification → counts and a new-candidate list of (a) URL-dup / (b) Path-dup / (c) Genuine new. **Only "Genuine new" are true candidates.**
• **Manual fallback** (when the tool is unavailable): (a) frontmatter `source:`/`url:`/`source_url:` → `by_url` match → SKIP, (b) on URL non-match, raw path (Unicode quotes normalized) → `by_path` match → SKIP, (c) only those that match neither and are not 0-byte are genuine. **No `by_path`-only prefilter** — a re-scrape varies the path, so the URL is the key (SoT: `prefilter_ingest.py`).
• Scan extensions: `.md` + `.pdf`
• Exclude under `raw/NewsScrap/Raindrop/` (already-ingested legacy)
• Empty files (0-byte) auto-skipped

Parallel Batch (Genuine new 3+)

If there are **3 or more** Genuine new candidates, branch out with the Agent tool. 12-step split:

• **Pre-fanout (main, once)**: read the wiki (`index.md`·`overview.md`) — shared sub-agent context
• **Fanout (N sub-agents in parallel)**: each sub-agent, for one candidate, (a) reads the raw, (b) matches URL/path, (c) writes `wiki/sources/<slug>.md`. It reports back to main with collected results (proposed new entities/concepts · `##