/rag-auditor

name: rag-auditor
description: 'Evaluates RAG pipeline quality across retrieval (precision, recall, MRR) and generation (groundedness, hallucination rate). Triggers on: "audit RAG pipeline", "RAG quality", "hallucination detection", "why is RAG failing", "grounding check". NOT for general architecture audits, use architecture-reviewer.'
metadata:
  version: 1.1.1
  category: review
  tags: [rag, retrieval, hallucination, grounding]
  difficulty: advanced
  phase: build

RAG Auditor

Systematic RAG pipeline evaluation across the full retrieval-generation chain: designs evaluation query sets, measures retrieval metrics (Precision@K, Recall@K, MRR), evaluates generation quality (groundedness, completeness, hallucination rate), diagnoses component-level failures, and recommends targeted improvements.

Reference Files

| File | Contents | Load When | | ---------------------------------- | -------------------------------------------------------------------------- | ---------------------------- | | `references/retrieval-metrics.md` | Precision@K, Recall@K, MRR, NDCG definitions and calculation | Always | | `references/generation-metrics.md` | Groundedness, completeness, hallucination detection methods | Generation evaluation needed | | `references/failure-taxonomy.md` | RAG failure categories: retrieval, generation, chunking, embedding | Failure diagnosis needed | | `references/diagnostic-queries.md` | Designing evaluation query sets, known-answer questions, difficulty levels | Evaluation setup |

Prerequisites

• Access to the RAG pipeline (or its outputs for post-hoc evaluation)
• A set of test queries with known-correct answers
• Understanding of the pipeline components (embedding model, retriever, generator)

Workflow

Phase 1: Pipeline Inventory

Document the RAG pipeline configuration:

1. **Document source** — What documents are indexed? Format, count, size. 2. **Chunking** — Strategy (fixed-size, semantic, paragraph), chunk size, overlap. 3. **Embedding** — Model name and version, dimensionality. 4. **Vector store** — Type (FAISS, Pinecone, Chroma, pgvector), index type. 5. **Retrieval** — Method (similarity, hybrid, reranking), top-K parameter. 6. **Generation** — Model, prompt template, context window usage.

Phase 2: Design Evaluation Queries

Create a diverse set of test queries:

| Query Type | Purpose | Count | | ---------------------- | ------------------------------------------- | ----- | | Known-answer (factoid) | Measure retrieval + generation accuracy | 10+ | | Multi-hop | Require combining info from multiple chunks | 5+ | | Unanswerable | Not in the corpus — should abstain | 3+ | | Ambiguous | Multiple valid interpretations | 3+ | | Recent/updated | Test freshness | 2+ |

For each query, document the expected answer and the source chunk(s).

Phase 3: Evaluate Retrieval

For each test query, measure:

1. **Precision@K** — Of the K retrieved chunks, how many are relevant? 2. **Recall@K** — Of all relevant chunks in the corpus, how many were retrieved? 3. **MRR (Mean Reciprocal Rank)** — How high is the first relevant chunk ranked? 4. **Chunk relevance** — Score each retrieved chunk: Relevant, Partially Relevant, Irrelevant.

Phase 4: Evaluate Generation

For each test query with retrieved context:

1. **Groundedness** — Is every claim in the response supported by the retrieved context? Score: 0 (hallucinated) to 1 (fully grounded). 2. **Completeness** — Does the response use all relevant information from the context? Score: 0 (ignored context) to 1 (complete). 3. **Hallucination detection** — Identify specific claims not supported by context. 4. **Abstention** — For unanswerable queries, does the model correctly say "I don't know"?

Phase 5: Diagnose Failures

For every incorrect or low-quality response, classify the root cause:

| Failure Type | Diagnosis | Indicator | | -------------------- | -------------------------------------------------- | ---------------------------------------- | | Retrieval failure | Relevant chunks not retrieved | Low Recall@K | | Ranking failure | Relevant chunk retrieved but ranked low | Low MRR, high Recall | | Chunk boundary issue | Answer split across chunk boundaries | Partial matches in multiple chunks | | Embedding mismatch | Query semantics don't match chunk embeddings | Relevant chunk has low similarity score | | Generation failure | Correct context but wrong answer | High retrieval scores, low groundedness | | Hallucination | Model invents facts not in context | Claims not traceable to any chunk | | Over-abstention | Model refuses to answer when context is sufficient | Unanswered with relevant context present |

Phase 6: Recommendations

Based on failure analysis, recommend specific improvements:

| Failure Pattern | Recommendation | | --------------------- | -------------------------------------------------------------- | | Chunk boundary issues | Increase overlap, try semantic chunking | | Low Precision@K | Reduce K, add reranking stage | | Low Recall@K | Increase K, try hybrid search | | Embedding mismatch | Try different embedding model, add query expansion | | Hallucination | Strengthen grounding instruction in p