Fixed-size Chunking

Why chunking matters

Most RAG systems retrieve documents at the chunk level, not the document level. Chunking is the process of splitting source documents into smaller pieces before embedding and indexing them. The quality of retrieval — and therefore the quality of the generated answer — depends heavily on how well your chunks represent coherent, self-contained pieces of information.

Fixed-size chunking splits text into chunks of N tokens (or characters) with M tokens of overlap between consecutive chunks. The overlap ensures that sentences or ideas that span a chunk boundary aren't cut in half — the context bleeds into the next chunk. It's the simplest strategy and a good starting point for any RAG system.

Fixed-size chunking implementation

from langchain.text_splitter import TokenTextSplitter, RecursiveCharacterTextSplitter

# Token-based chunking (recommended — aligns with model context)
splitter = TokenTextSplitter(
    chunk_size=512,       # tokens per chunk
    chunk_overlap=50,     # overlap between consecutive chunks
    encoding_name="cl100k_base",  # tiktoken encoding (GPT-4, ada-002)
)
chunks = splitter.split_text(document_text)

# Character-based chunking (simpler, language-agnostic)
char_splitter = RecursiveCharacterTextSplitter(
    chunk_size=2000,      # characters per chunk (rough ~500 token equiv)
    chunk_overlap=200,
    separators=["

", "
", ". ", " ", ""],  # try to split at boundaries
)
chunks = char_splitter.split_documents(documents)

print(f"Document: {len(document_text)} chars → {len(chunks)} chunks")
for i, chunk in enumerate(chunks[:3]):
    print(f"Chunk {i}: {len(chunk)} chars | {chunk[:80]}...")

Choosing chunk size and overlap

The right chunk size depends on your embedding model's context window and the nature of your documents:

• Small chunks (128–256 tokens): High precision retrieval — each chunk contains a focused idea. But individual chunks may lack context to answer questions. Good for: FAQs, structured data, short-answer retrieval.
• Medium chunks (512–1024 tokens): The standard sweet spot. Captures full paragraphs with surrounding context. Good for: general documents, articles, documentation.
• Large chunks (2048+ tokens): Low precision but captures more context per chunk. Useful when you pass the full chunk to the LLM without further trimming. Good for: legal documents, long-form analysis where paragraphs are interdependent.

Overlap: A 10–15% overlap is usually sufficient. For 512-token chunks, use 50–75 tokens overlap. Too much overlap wastes index space; too little causes missed context at boundaries.

Token vs character chunking

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def token_chunker(text: str, chunk_size: int = 512, overlap: int = 50):
    tokens = enc.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))
        start += chunk_size - overlap
    return chunks

# Compare sizes
text = "A" * 10000  # 10K chars
char_chunks = [text[i:i+2000] for i in range(0, len(text), 1800)]
token_chunks = token_chunker(text, chunk_size=512, overlap=50)

print(f"Character chunks: {len(char_chunks)}")
print(f"Token chunks: {len(token_chunks)}")
# Character count per chunk is unpredictable for multilingual text
# Token count per chunk is exact — use for LLM context budgeting

Use token-based chunking when you need predictable context window usage. Character-based is faster to compute and language-agnostic, but token counts vary across languages (CJK characters ≈ 1 char : 1-2 tokens, ASCII ≈ 4 chars : 1 token).

Benchmarks vs other strategies

On standard RAG benchmarks (RAGAS, TruLens), fixed-size chunking at 512 tokens with 50-token overlap typically achieves 70–85% of the performance of more sophisticated strategies (semantic chunking, sentence-window). For many production use cases, the simplicity and speed advantage of fixed-size chunking outweighs the quality gap.

Fixed-size chunking breaks down when: documents have highly variable density (a 512-token chunk of legal boilerplate contains less information than 512 tokens of dense technical content), sentences span chunk boundaries (partially captured sentences hurt both embedding quality and LLM coherence), and documents have natural semantic units (chapters, sections) that should stay together.

Recommendation: start with fixed-size 512 tokens + 50 overlap, measure retrieval quality, then try sentence-window or semantic chunking if you need improvement.

Integration with vector stores

from langchain.text_splitter import TokenTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
import os

# Load and chunk documents
splitter = TokenTextSplitter(chunk_size=512, chunk_overlap=50)
chunks = splitter.create_documents(
    texts=[doc.text for doc in documents],
    metadatas=[{"source": doc.source, "page": doc.page} for doc in documents],
)

# Embed and store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
)

# Query
results = vectorstore.similarity_search("What is attention?", k=4)
for r in results:
    print(f"[{r.metadata['source']}] {r.page_content[:200]}")

Gotchas

Mid-sentence splits: Fixed-size chunking can cut sentences in half. Use RecursiveCharacterTextSplitter with sentence-boundary separators to reduce this. Overlap also helps — the end of chunk N and start of chunk N+1 overlap, so a query that needs the full sentence can retrieve it from one of them.

Table and code chunking: Fixed-size chunking destroys table structure and code blocks. For documents with tables, use a dedicated table parser. For codebases, use AST-based chunking (split at function/class boundaries) rather than fixed-size.

Chunk size ≠ embedding window: Ensure your chunk_size fits within the embedding model's token limit. text-embedding-3-small handles 8191 tokens, but smaller chunk sizes (256-512) often yield better retrieval precision because the embedding captures a more focused concept.

Re-chunking after document updates: When a source document is updated, you need to re-chunk and re-embed the entire document. There's no "partial update" for chunks — track document version hashes to detect changes and trigger re-ingestion.