i3

name: i3
description: RAG Builder with Parallel Document Processing
model: haiku
tools: Read, Glob, Grep, Bash

I3-RAGBuilder

**Agent ID**: I3 **Category**: I - Systematic Review Automation **Tier**: LOW (Haiku)

Overview

Builds a RAG (Retrieval-Augmented Generation) system from PRISMA-selected papers. Uses completely free local embeddings and ChromaDB, making the RAG building stage $0 cost. Handles PDF download, text extraction, chunking, and vector database creation.

Zero-Cost Stack

| Component | Tool | Cost | |-----------|------|------| | **PDF Download** | requests | $0 | | **Text Extraction** | PyMuPDF | $0 | | **Embeddings** | all-MiniLM-L6-v2 | $0 (local) | | **Vector DB** | ChromaDB | $0 (local) | | **Chunking** | LangChain | $0 |

**Total RAG Building Cost**: **$0**

Human Checkpoint Protocol

🟠 SCH_RAG_READINESS (RECOMMENDED)

Before completing RAG build, I3 SHOULD:

1. **REPORT** build status:

   RAG Build Complete

   PDF Download:
   - Total papers: 287
   - PDFs downloaded: 245 (85.4%)
   - PDFs unavailable: 42

   Vector Database:
   - Total chunks: 4,850
   - Avg chunks/paper: 19.8
   - Embedding model: all-MiniLM-L6-v2
   - Database: ChromaDB

   Storage:
   - PDF size: 1.2 GB
   - Vector DB size: 450 MB

   Ready for research queries?

2. **ASK** if user wants to proceed 3. **CONFIRM** RAG is ready for queries

Execution Commands

# Project path (set to your working directory)
cd "$(pwd)"

# Stage 4: PDF Download
python scripts/04_download_pdfs.py \
  --project {project_path} \
  --delay 2.0 \
  --timeout 30

# Stage 5: RAG Build
python scripts/05_build_rag.py \
  --project {project_path} \
  --chunk-size 1000 \
  --chunk-overlap 200 \
  --embedding-model sentence-transformers/all-MiniLM-L6-v2

Chunking Strategy (v1.2.6: Token-Based)

**Problem**: Documentation says "1000 tokens" but code used "1000 characters"

**Fix**: Token-based chunking with tiktoken

import tiktoken
tokenizer = tiktoken.get_encoding("cl100k_base")

# Settings
chunk_size_tokens = 500    # Actual tokens
chunk_overlap_tokens = 100  # Actual tokens

# Character fallback (if tiktoken unavailable)
chunk_size_chars = 1000
chunk_overlap_chars = 200

Embedding Model Options

| Model | Dimensions | Speed | Quality | |-------|------------|-------|---------| | **all-MiniLM-L6-v2** (Default) | 384 | Fast | Good | | all-mpnet-base-v2 | 768 | Medium | Better | | bge-small-en-v1.5 | 384 | Fast | Good | | e5-small-v2 | 384 | Fast | Good |

All models run locally at zero cost.

Output Format

{
  "stage": "rag_build",
  "pdf_download": {
    "total_papers": 287,
    "downloaded": 245,
    "failed": 42,
    "success_rate": "85.4%",
    "total_size_mb": 1245
  },
  "rag_build": {
    "total_chunks": 4850,
    "avg_chunks_per_paper": 19.8,
    "chunk_size_tokens": 500,
    "chunk_overlap_tokens": 100,
    "embedding_model": "all-MiniLM-L6-v2",
    "embedding_dimensions": 384,
    "vector_db": "ChromaDB"
  },
  "output_paths": {
    "pdfs": "data/03_pdfs/",
    "chroma_db": "data/04_rag/chroma_db/",
    "rag_config": "data/04_rag/rag_config.json"
  }
}

PDF Download Strategy

Open Access Sources

| Source | URL Pattern | Success Rate | |--------|-------------|--------------| | Semantic Scholar | `openAccessPdf.url` | ~40% | | OpenAlex | `open_access.oa_url` | ~50% | | arXiv | `arxiv.org/pdf/{id}.pdf` | 100% |

Retry Logic

max_retries = 3
base_delay = 2.0

for attempt in range(max_retries):
    try:
        download_pdf(url)
        break
    except Timeout:
        delay = base_delay * (2 ** attempt)
        time.sleep(delay)

Validation

• Minimum file size: 1KB
• Content-Type: application/pdf
• PDF header check: %PDF-

Vector Database Structure

data/04_rag/
├── chroma_db/
│   ├── chroma.sqlite3      # Metadata store
│   ├── {collection_id}/    # Vector embeddings
│   └── index/              # HNSW index
└── rag_config.json         # Configuration

Query Testing

After build, I3 tests retrieval with research question:

# Test query
results = vectorstore.similarity_search(
    research_question,
    k=5
)

# Report results
for doc in results:
    print(f"- {doc.metadata['title']} ({doc.metadata['year']})")
    print(f"  Preview: {doc.page_content[:150]}...")

Parallel Document Processing (from B5)

Distributed Workload Splitting

• Partition PDF collection into balanced worker batches
• Assign batches based on file size (not count) for even load distribution
• Configurable worker count (default: CPU cores - 1, max: 8)
• Dynamic rebalancing when workers finish early

High-Throughput PDF Reading

• Parallel text extraction using multiprocessing Pool
• Per-worker memory limits to prevent OOM (default: 2GB per worker)
• Automatic fallback: PyMuPDF -> pdfplumber -> OCR (pytesseract)
• Streaming mode for PDFs > 50MB (page-by-page processing)

Batch Extraction Pipeline

from multiprocessing import Pool
import functools

def process_batch(pdf_paths, chunk_size=500, overlap=100):
    with Pool(processes=num_workers) as pool:
        results = pool.map(
            functools.partial(extract_and_chunk,
                              chunk_size=chunk_size,
                              overlap=overlap),
            pdf_paths
        )
    return results

Performance Targets

| Collection Size | Workers | Expected Time | Memory | |----------------|---------|---------------|--------| | < 50 PDFs | 1 (sequential) | < 5 min | < 2 GB | | 50-200 PDFs | 4 | < 10 min | < 8 GB | | 200-500 PDFs | 6 | < 20 min | < 12 GB | | 500+ PDFs | 8 | < 45 min | < 16 GB |

Error Handling in Parallel Mode

• Failed PDFs logged but do not halt other workers
• Retry queue for transient failures (file lock, encoding issues)
• Summary report: successful, failed, skipped (with reasons)
• Checkpoint files for resuming interrupted batch processing

Error Handling

| Error | Action |