/agentsop-idempotent-ingestion

name: agentsop-idempotent-ingestion
version: 0.1.0
description: |
  Re-ingest-correctness SOP for production RAG. Activate when a calling agent
  builds, reviews, or debugs an ingestion pipeline that runs more than once over
  a changing corpus — scheduled re-index, incremental updates, CI re-ingest, or
  a "retrieval has duplicates / shows deleted docs" bug. Encodes the rule —
  **ingestion must be idempotent: a document's content hash decides
  insert/update/skip, so re-running over unchanged docs is a no-op** — plus the
  docstore + doc-hash upsert machinery (LlamaIndex `IngestionPipeline` +
  `DocstoreStrategy`), the delete-propagation problem, and cross-framework
  equivalents (LangChain `index()` + `RecordManager`, manual hash table). ENHANCE
  overlay over [[llamaindex]]: the IngestionPipeline exists in the base skill but
  the re-ingest-correctness contract is not surfaced.
trigger_keywords:
  - "idempotent ingestion"
  - "re-ingest"
  - "re-index"
  - "docstore"
  - "doc hash"
  - "upsert"
  - "duplicate chunks"
  - "IngestionPipeline"
  - "RecordManager"
  - "incremental update"
  - "scheduled reindex"
when_to_use:
  - "any ingestion pipeline that will run more than once (cron, CI, webhook, manual re-run)"
  - "a corpus where source documents are added, edited, or deleted over time"
  - "debugging duplicate / stale chunks polluting retrieval after a re-run"
  - "reviewing a PR that calls VectorStoreIndex.from_documents / pipeline.run without a docstore"
  - "designing the ingestion side of a production RAG before first deploy"
when_not_to_use:
  - "a one-shot index built once and never refreshed (static corpus, throwaway notebook)"
  - "the corpus is small enough to fully rebuild from scratch in seconds and rebuild-on-every-run is genuinely acceptable (measure first)"
  - "pure retrieval / query-time work with no ingestion path"

Idempotent Ingestion · Re-Ingest-Correctness SOP

> Third-person operating model for a coder agent that owns ingestion > correctness across repeated runs. The audience is the LLM agent writing or > reviewing the pipeline code — not the end user.

> **One sentence**: *The interesting run is the second one. A correct pipeline > hashes each document, looks the hash up in a docstore, and inserts / updates > / skips accordingly — so re-running over unchanged docs changes nothing.*

This skill is an **ENHANCE overlay** over [[llamaindex]]. The base skill names `IngestionPipeline` with `docstore` + `UPSERTS_AND_DELETE` as a hardening must-have (OP-08, anti-pattern A10) but stops at "use it". This overlay surfaces the *correctness contract* — what idempotency actually means, how the hash decides the action, and how it generalises beyond LlamaIndex.

---

1. 何时激活 (Activation Rules)

Activate this skill whenever **any** of the following holds:

1. The codebase contains an ingestion call (`VectorStoreIndex.from_documents`, `pipeline.run(documents=...)`, `vectorstore.add_documents`, a custom embed

• upsert loop) **and** that call will execute more than once over a corpus

that can change between runs. 2. The user mentions any of: re-ingest, re-index, scheduled / nightly index refresh, incremental document updates, docstore, doc hash, upsert, `RecordManager`, "keep the index in sync with the source". 3. A bug report says "I see the same answer chunk twice", "retrieval returns duplicates", "deleted a file but it still shows up in answers", "the index keeps growing every night even though nothing changed". 4. PR review: new ingestion code that builds the index with no docstore / no hash-keyed dedup and the corpus is live (LlamaIndex base skill A10). 5. A production RAG is about to ship and the ingestion side has only ever been tested as a cold first run.

Do **not** activate when:

• The index is built **once** from a static corpus and never refreshed — there

is no second run to make idempotent.

• The corpus is tiny and rebuilding from scratch each run is *measurably*

cheaper than maintaining a docstore (rare; verify, don't assume).

• The task is purely query-time (retrieval, rerank, synthesis) with no

ingestion path in scope.

---

2. 核心心智模型 (Core Mental Model)

Three principles. Violating any one means a second run can duplicate or corrupt the index — regardless of how good the retrieval stack downstream is.

Principle 1 — Ingestion must be idempotent; the hash is the decision

> **Re-running the pipeline over an unchanged document is a no-op.** Run it > once, run it a thousand times — same index. The mechanism: each document is > reduced to a stable **content hash** (and a stable **doc id**); that hash is > looked up in a **docstore** (a key→hash record store separate from the vector > store); the lookup result decides the action.

            doc_id seen?           hash changed?
                │                       │
   ┌────────────┴───────────┐    ┌──────┴──────┐
   no                       yes  no            yes
   │                         │   │              │
INSERT (embed + upsert)      │  SKIP        UPDATE
                             └──────────────►(delete old chunks,
                                              re-embed, upsert)

The hash is the whole game. Without it, the pipeline cannot tell "I have seen this exact content" from "this is new" — so it re-embeds and re-adds everything, and the vector store accumulates duplicates. (LlamaIndex base skill failure #3 / A10; `developers.llamaindex.ai/.../loading/ingestion_pipeline/`.)

Principle 2 — The docstore is a separate ledger, not the vector store

The vector store holds embeddings keyed by node id. The **docstore** holds, per document, `{doc_id → content_hash}` (and the node ids derived from it). They are two stores with two jobs:

| Store | Holds | Answers | |---|---|---| | Docstore | `doc_id → hash`, doc→node mapping | "have I seen this content before?" | | Vector store | `node_id → embedding + metadata` | "what is semantically ne