RAG Systems

What Is RAG?

RAG (Retrieval-Augmented Generation) is a pattern where an LLM generates answers using external knowledge: retrieve relevant documents from a corpus → feed them to the LLM as context → the LLM generates grounded responses. This solves two critical problems: hallucination (LLMs making up facts) and knowledge cutoff (LLMs frozen at training time).

Unlike fine-tuning (expensive, permanent, requires retraining), RAG allows you to update knowledge by simply adding new documents to the corpus. The LLM sees fresh context at query time, making RAG ideal for domains where facts change frequently: legal documents, medical records, product catalogs, financial reports.

When RAG Makes Sense

RAG Pipeline Stages

A production RAG system has clear stages: offline indexing (happens once or in batch) and online serving (happens per-query). Separating these allows optimization: indexing can be slow/expensive, serving must be fast.

Offline Indexing

Document ingestion: Load PDFs, HTML, databases, Slack dumps, etc. Parsing: Extract text, tables, images. Preserve structure. Chunking: Split into retrieval-sized pieces (typically 256–1024 tokens). Enrichment: Add metadata (source, date, author), summaries, keywords. Embedding: Convert chunks to dense vectors. Storage: Index in vector DB + maintain metadata store.

Online Serving

Query embedding: Convert user query to vector using same embedding model. Retrieval: Semantic search in vector DB → top-k candidates. Optional re-ranking: Cross-encoder scores candidates by relevance → top-n. Context assembly: Grab full documents or passages. Prompt construction: Format context + query for LLM. Generation: LLM reads context and generates grounded answer. Citations: Track which documents supported answer.

Framework Comparison: Haystack vs LlamaIndex vs LangChain

Three mature frameworks dominate the RAG ecosystem. Each has different philosophy, maturity, and strengths. Choice depends on your team's background and deployment target.

Haystack (Production-Grade RAG)

Originally built at Deepset specifically for RAG. Strong document pipeline, excellent for multi-document workflows. Pipelines are explicit DAGs you can visualize. Great for teams committed to RAG as core feature. Good OSS community, commercial support available.

LlamaIndex (Query Translation)

Focused on indexing diverse data sources: PDFs, databases, APIs, Notion. Excels at translating high-level queries into data-specific access patterns. Lightweight. Great for prototyping. Growing agent support. Simpler than Haystack for beginners.

LangChain (General Purpose)

Broadest tool ecosystem. Not RAG-specific — emphasizes chains (workflows), agents (tool use), memory. Better for applications that mix RAG with other LLM capabilities (code generation, arithmetic, APIs). Largest community. Heavy on abstractions, can be opinionated.

Chunking Strategies: The Foundation

Chunking is arguably the single most important decision in RAG. Bad chunking breaks retrieval; good chunking makes everything else easy. Chunk size affects recall (smaller = more docs retrieved), precision (larger = fewer false positives), and latency (more chunks = slower embedding).

Fixed-Size Chunking

Split every N tokens, optionally with overlap. Simplest approach. Deterministic. Problem: breaks mid-sentence, splits concepts. Best for diverse corpora where structure isn't known.

Semantic Chunking

Split at sentence or paragraph boundaries, then merge until chunk reaches target size. Respects semantics. Better coherence. Slightly slower at index time. Recommended for structured documents: reports, articles, wikis.

Hierarchical (Parent-Child)

Index small chunks for precision, but fetch parent chunk for context. Example: 256-token chunks for retrieval, but show 1024-token parent to LLM. Combines precision and context. Requires document hierarchy awareness (sections, chapters).

Snippet Size Guidelines

Embedding Model Selection

Your embedding model determines what "semantic similarity" means to your RAG system. Choice affects both quality and cost. General-purpose models (OpenAI, Anthropic) are good defaults. Domain-specific models (legal, medical, code) can be better if you have the budget.

Popular Embedding Models

Embedding Quality vs Cost

High quality (proprietary): OpenAI text-embedding-3-large. Better generalization, higher cost. Balanced (open): BAAI/bge-large. Strong performance, free, can self-host. Fast (lightweight): e5-base-v2, all-MiniLM-L6. Lower latency, smaller models, acceptable quality.

Vector Database Choice

Vector DBs store high-dimensional embeddings and answer nearest-neighbor queries fast. Dozens exist; choice depends on scale, latency budget, and whether you need hybrid (dense+sparse) search.

Vector Database Comparison

Key Considerations

Hybrid search: If your corpus has domain-specific terminology (legal, medical, code), combine dense embeddings with BM25. Weaviate, Qdrant, Elasticsearch all support this. Scale: Small corpus (<100k docs) → Chroma is fine. Large corpus (millions) → Pinecone or self-hosted Qdrant/Milvus. Latency: Managed (Pinecone) is simpler ops; self-hosted gives you control. Cost: Chroma/Milvus/Qdrant OSS are free; Pinecone charges per query.

Enterprise RAG Considerations

Moving RAG from prototype to production requires handling scale, reliability, and governance that lab settings don't require. Enterprise RAG introduces access control, data freshness, multi-tenancy, and auditability.

Access Control & Multi-Tenancy

In enterprise, different users see different documents. A single shared vector DB can't enforce this naively. Solutions: document-level filtering (tag chunks with user_id, filter at retrieval time), per-user indexes (overkill for most cases), row-level security in vector DB (Qdrant, Weaviate support metadata-based filtering). Recommended: tag chunks with accessible_users and filter before ranking.

Data Freshness

How often do you re-index? Daily/weekly is typical; minutes requires streaming indexing (harder). Strategy: batch indexing (new docs → index offline → swap at midnight), incremental indexing (new docs → add to existing index immediately), document versioning (keep old chunks, mark as stale, weight new higher). For real-time data (stock prices, live scores), RAG isn't the right tool alone.

Observability & Audit

What was retrieved? What was generated? Did we hallucinate? Track: queries and results (audit trail), retrieval quality (MRR, NDCG metrics), answer quality (user feedback, LLM-based scoring), latency per stage (retrieve, rerank, generate). Log sources of answers for compliance.

Best Practices Checklist

References and Related Concepts

• Haystack — Production RAG framework by Deepset
• Cognee — Graph-based knowledge management for RAG

• Advanced RAG Patterns — Query transformation, reranking, multi-hop
• Retrieval Technologies — Dense, sparse, and hybrid search
• Vector Databases — Storage and indexing for embeddings
• Embeddings — Vector representations for semantic search
• Chunking Strategies — Document splitting for retrieval

• Paper Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv:2005.11401
• Docs Haystack Documentation. docs.haystack.deepset.ai ↗
• Docs LlamaIndex Documentation. docs.llamaindex.ai ↗
• Blog Embedding Quality & Model Selection. MTEB Leaderboard ↗