/ingest-sources

name: ingest-sources
description: Process multiple source documents with Extract-Then-Aggregate discipline. Use when user shares multiple transcripts, emails, or documents for batch processing. See also: `capture-meeting` for a single meeting transcript; `file-document` for a single document; `summarize-doc` when you only need a summary.
argument-hint: [folder-path]
effort-level: max

Ingest Sources

Process multiple source documents (transcripts, emails, documents) using Extract-Then-Aggregate discipline to ensure no entity with dedicated sources gets lost.

Trigger

• "Process these transcripts"
• "Here are my notes from [event]"
• Multiple files shared in sequence
• "Here's everything about [topic]"
• Folder path provided with multiple files
• `/ingest-sources`

Why This Skill Exists

When processing many sources, the failure mode is jumping to aggregation and missing entities that have dedicated sources but aren't prominent in high-traffic threads. A person with 2 transcripts dedicated to them can get lost if they're not mentioned often in emails.

**The discipline:** Inventory before processing, extraction before synthesis.

Input

User provides one of:

• Folder path containing multiple files
• List of file paths
• Multiple documents pasted in sequence
• Reference to previously shared content

The Five-Phase Workflow

Phase 1: Inventory

**Before reading any content**, create a manifest of all sources:

| # | Filename | Type | Date | Size | Likely Entities |
|---|----------|------|------|------|-----------------|
| 1 | call-with-sarah.md | transcript | 2026-01-15 | 4.2KB | Sarah Chen |
| 2 | jim-partnership-email.md | email | 2026-01-16 | 1.8KB | Jim Ferry |
| 3 | acme-contract.pdf | document | 2026-01-17 | 52KB | Acme Corp |
...

**Summary:**
- Total: 36 sources
- Date range: Jan 15 - Feb 1
- Types: 28 transcripts, 5 emails, 3 documents

Show inventory to user before proceeding. This prevents partial processing.

Phase 2: File-Then-Extract (Per Document)

**CRITICAL:** For each document, file it BEFORE extracting. This ensures provenance.

For each source in inventory:
    1. READ the full content
    2. CALL `claudia memory document store --project-dir "$PWD"` immediately (do not skip!)
    3. THEN extract entities/facts/commitments

Process each document systematically. Use `IngestService` (via local Ollama) when available, or extract directly.

**Auto-detect source type:**

• `.md`, `.txt` with participant names → `meeting` mode
• Email headers detected → `email` mode
• `.pdf` or formal structure → `document` mode
• Mixed content → `general` mode

**Extraction schema per document:**

Source #1: call-with-sarah.md
├── entities[]
│   ├── name: "Sarah Chen"
│   ├── type: person
│   ├── mention_count: 47
│   └── first_context: "Product lead at Acme Corp"
├── facts[]
│   ├── content: "Sarah prefers async communication"
│   ├── about: ["Sarah Chen"]
│   └── importance: 0.7
├── commitments[]
│   ├── content: "Send proposal by Friday"
│   ├── who: "user"
│   ├── to: "Sarah Chen"
│   └── deadline: "2026-02-07"
├── relationships[]
│   ├── source: "Sarah Chen"
│   ├── target: "Acme Corp"
│   └── relationship: "works_at"
└── dedicated_to: "Sarah Chen"  ← CRITICAL: This source is primarily ABOUT Sarah

**Progress tracking:**

Extracting: [========>   ] 28/36 (78%)

**The `dedicated_to` field is essential.** If a source is primarily about a specific entity (not just mentioning them), mark it. This prevents the "missing entity" problem.

Phase 3: Consolidation

After all extractions complete, merge by entity:

**Canonicalize names:**

• Check existing `entity_aliases` table for known aliases
• Fuzzy match "Sarah" vs "Sarah Chen" vs "S. Chen"
• Ask user to confirm ambiguous matches

**Merge semantically identical facts:**

• "Sarah prefers Slack" + "Sarah likes async comms" → single fact about communication preference
• Keep the more specific version

**Track source counts:**

Entity: Sarah Chen
├── Dedicated sources: 4 (#1, #5, #12, #18)
├── Total mentions: 12 sources
├── Facts extracted: 8
└── Commitments: 2

Phase 4: Verification

**Before storing anything**, verify completeness:

### Entity Coverage

| Entity | Dedicated Sources | Total Mentions | Sources |
|--------|-------------------|----------------|---------|
| Sarah Chen | 4 | 12 | #1, #5, #12, #18, ... |
| Jim Ferry | 2 | 6 | #2, #15, ... |
| Acme Corp | 3 | 8 | #3, #7, #22, ... |
| Project Alpha | 0 | 4 | #4, #8, #11, #19 |

### Dedicated Source Rule

**Any entity with 2+ dedicated sources MUST appear proportionally in the final output.**

If Jim Ferry has 2 transcripts dedicated to him but doesn't show up in the entity coverage summary, that's a verification failure. Stop and investigate.

### Gaps Detected

- Source #14: No entities extracted (may need manual review)
- Source #22: References "the investor" without name

### Completeness Check

Before proceeding:
- [ ] Every dedicated source entity appears in coverage
- [ ] No sources skipped or failed
- [ ] Ambiguous entity names resolved
- [ ] Gaps acknowledged or explained

**User must confirm** before proceeding to storage. This is the checkpoint that catches the "missing entity" problem.

Phase 5: Storage

After user confirms verification:

**1. Verify all sources filed:** Sources were already filed during Phase 2 (File-Then-Extract). Verify the file count matches:

Confirm: [N] sources filed to ~/.claudia/files/

If any sources weren't filed in Phase 2, file them now before proceeding.

Files are auto-routed to entity folders:

• `people/sarah-chen/transcripts/...`
• `clients/acme-corp/documents/...`
• `projects/alpha/emails/...`

**2. Create/update entities:**

claudia memory batch --project-dir "$PWD" <<'EOF'
[
  { "op": "entity", "name": "Sarah Chen", "type": "person", "description": "Product lead at Acme Corp" },
  { "op": "entity", "name": "Jim Ferry", "type": "person", "des