/agentsop-hybrid-retrieval

name: agentsop-hybrid-retrieval
version: 0.1.0
description: |
  Enhancement-overlay SOP for adding sparse (BM25 / keyword) retrieval alongside
  dense (embedding) retrieval. Activate when a calling agent is building, reviewing,
  or debugging a retrieval pipeline whose corpus contains exact-match tokens —
  identifiers, error codes, SKUs, API/function names, proper nouns, citations, rare
  jargon — that pure dense embedding silently misses. Encodes the single decision
  rule (**hybrid is traffic-driven, not theoretical: add sparse only when the query
  share that depends on exact tokens is non-trivial**), the wiring of
  QueryFusionRetriever-style fusion (RRF vs alpha-weighted), and per-query-type alpha
  tuning. Frame the work as recovering lexical identity that dense pooling destroys,
  not as "add keyword search for completeness". Cross-links [[llamaindex]].
overlay: true
cross_links: [llamaindex, langchain]
trigger_keywords:
  - "hybrid search"
  - "hybrid retrieval"
  - "BM25"
  - "sparse retrieval"
  - "dense plus sparse"
  - "QueryFusionRetriever"
  - "EnsembleRetriever"
  - "RRF"
  - "reciprocal rank fusion"
  - "alpha tuning"
  - "keyword search RAG"
when_to_use:
  - "the corpus contains exact-match tokens: identifiers, error codes, SKUs, part numbers, API/function names, proper nouns, legal/medical citations, rare jargon"
  - "users report 'I searched the exact code/name and got nothing / the wrong doc' on a dense-only retriever"
  - "reviewing a retriever where traffic includes lexical-identity lookups but only embeddings are wired"
  - "tuning recall on a corpus where both meaning AND exact strings matter"
  - "deciding between pure dense, hybrid, or sparse-first for a new RAG corpus"
when_not_to_use:
  - "traffic is purely semantic / conceptual with <5% lexical-identity queries — hybrid is over-engineering"
  - "the corpus has no stable identifiers and queries never reference exact strings"
  - "pre-baseline: ship dense-only and measure first; hybrid is a Stage-3 optimization (see [[llamaindex]])"

Hybrid Retrieval · Dense + Sparse SOP

> Third-person operating model for a coder agent that owns retrieval recall on a > corpus where *both* meaning and exact tokens matter. The audience is the LLM > agent writing or reviewing retrieval code — not an end user.

> **One sentence**: *Dense captures meaning, sparse captures exact tokens; hybrid > wins when both matter — but only fuse them when traffic actually carries > exact-match queries, and tune the blend per query type or hybrid loses to dense.*

---

1. 何时激活 (Activation Rules)

Activate this skill when **any** of the following holds:

1. The corpus contains **exact-match tokens** that a query may reference verbatim: error codes (`ERR_SSL_PROTOCOL`), SKUs / part numbers (`A1-2293-X`), API or function names (`as_query_engine`), proper nouns, legal/medical citations (`42 U.S.C. § 1983`), version strings, rare jargon, ticket IDs. 2. A bug report says **"I searched the exact code/name/string and got nothing"**, or "the right document exists but dense retrieval ranks it below fuzzy near-misses". 3. PR review surfaces a retriever serving lexical-identity traffic but wired **dense-only** (`index.as_retriever(...)` / `similarity_search(...)` with no sparse leg). 4. You are tuning recall and have already exhausted the cheap dense knobs (prompt, embedding model, chunk size) per the [[llamaindex]] optimization ladder — hybrid is the next rung. 5. The user mentions hybrid search, BM25, sparse retrieval, RRF, `QueryFusionRetriever`, `EnsembleRetriever`, or vector-store-native hybrid (Qdrant/Weaviate/Pinecone).

Do **not** activate when:

• Traffic is purely semantic ("what does X mean?", "summarize the policy") with a

lexical-identity share **<5%** — adding BM25 doubles index footprint for no gain.

• The corpus has no stable identifiers and no query ever quotes an exact string.
• No dense baseline + eval loop exists yet. Hybrid is a **Stage-3** optimization

([[llamaindex]] Stage 3 step 4): baseline and measure before fusing.

---

2. 核心心智模型 (Core Mental Model)

Three principles. Violating any of them is why teams "try hybrid and conclude it didn't help".

Principle 1 — Dense and sparse fail in opposite directions

Dense embedding models **destroy lexical identity by pooling token representations**: querying a specific error string yields a vector that captures *"document about SSL errors"* rather than *"document containing this exact string"*. BM25 does the inverse — it scores against an **inverted index of exact tokens** and is blind to synonyms and paraphrase. (TianPan, *Hybrid search in production*, 2026; cited in [[llamaindex]] Dilemma 2.)

> **Operational corollary**: the symptom "I pasted the exact code and got nothing" > is not a bug in the embedding model — it is the embedding model working as > designed. The fix is a second retriever that indexes tokens, not a better > embedding.

Principle 2 — Hybrid is traffic-driven, not theoretical

Whether to add sparse is decided by the **query-type distribution of real traffic**, not by a belief that "more retrievers = better". The decision threshold is the **lexical share**: the fraction of queries whose correct answer hinges on an exact token. Below ~5% → dense-only. 5–50% → hybrid. Above ~50% (code, logs, legal) → invert to sparse-first with dense as a rerank signal. (Cited in [[llamaindex]] OP-04 / Dilemma 2.)

Principle 3 — One global alpha underperforms; tune per query type

`alpha` is the dense↔sparse blend (`alpha=1` → pure dense, `alpha=0` → pure sparse). A semantic query wants high alpha; a lexical-identity query wants low alpha. A single **global** alpha picked to help lexical queries *hurts* the semantic slice — which is exactly why a flat alpha "loses to pure dense" and teams wrongly conclude hybrid failed. Tune alpha **per query type**, or route per type and pick alpha per route. (LlamaIndex alpha-tuning blog; [[llamain