Context Compression

The noise problem in RAG context

You retrieve the top-5 chunks. Each chunk is 512 tokens. But the user's question is about a single sentence in chunk 3 — the other 4 chunks and 90% of chunk 3 are noise. You're sending 2560 tokens to the LLM when 50 tokens would suffice.

Two problems follow: cost (you pay for all 2560 tokens) and quality ("lost in the middle" — LLMs are less accurate when the relevant content is buried deep in a long context). Context compression solves both by stripping out the irrelevant parts before they reach the LLM.

Extractive compression

Extract only the sentences or fragments from the retrieved chunks that are relevant to the query, discarding the rest:

import anthropic

client = anthropic.Anthropic()

def extract_relevant_sentences(query: str, passage: str) -> str:
    '''Extract only the sentences from passage that answer the query.'''
    response = client.messages.create(
        model="claude-3-5-haiku-20241022",
        max_tokens=512,
        messages=[{"role": "user", "content": f'''
Extract only the sentences from the passage below that are directly relevant
to answering the question. If no sentences are relevant, return "IRRELEVANT".
Return only the extracted sentences, nothing else.

Question: {query}

Passage:
{passage}
'''}]
    )
    result = response.content[0].text.strip()
    return "" if result == "IRRELEVANT" else result

# Usage
passage = '''
Our company was founded in 2015 by Jane Smith.
Returns are accepted within 30 days of purchase with the original receipt.
We have offices in New York, London, and Singapore.
Refunds are processed to the original payment method within 5 business days.
Our CEO has over 20 years of retail experience.
'''
compressed = extract_relevant_sentences("How long does a refund take?", passage)
print(compressed)
# "Refunds are processed to the original payment method within 5 business days."

LLM-based compression

Instead of extracting sentences, ask the LLM to summarise the relevant portions:

import anthropic

client = anthropic.Anthropic()

def compress_to_answer(query: str, passages: list[str]) -> str:
    '''Compress multiple retrieved passages into a focused context.'''
    combined = "\n\n---\n\n".join(passages)
    response = client.messages.create(
        model="claude-3-5-haiku-20241022",
        max_tokens=300,
        messages=[{"role": "user", "content": f'''
Given the following retrieved passages and a question, extract and condense
only the information relevant to answering the question.
Be concise. If a passage has no relevant information, skip it.

Question: {query}

Retrieved passages:
{combined}

Condensed relevant information:
'''}]
    )
    return response.content[0].text.strip()

query = "What is the refund policy?"
passages = [
    "Our company was founded in 2015. We have 500 employees globally.",
    "Returns are accepted within 30 days. Refunds take 5 business days.",
    "We ship worldwide. Free shipping on orders over $50.",
]
compressed = compress_to_answer(query, passages)
print(compressed)
# "Returns are accepted within 30 days of purchase. Refunds are processed in 5 business days."

LangChain ContextualCompressionRetriever

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor
from langchain_anthropic import ChatAnthropic
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain.schema import Document

# Base retriever
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
docs = [
    Document(page_content="Founded 2015, 500 employees globally."),
    Document(page_content="Returns within 30 days. Refunds in 5 business days."),
    Document(page_content="Ships worldwide. Free shipping over $50."),
]
vectorstore = Chroma.from_documents(docs, embeddings)
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

# Compression layer
llm = ChatAnthropic(model="claude-3-5-haiku-20241022")
compressor = LLMChainExtractor.from_llm(llm)

# Combine: retrieves 5 docs, compresses each, returns only relevant snippets
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever
)

compressed_docs = compression_retriever.invoke("What is the refund timeline?")
for doc in compressed_docs:
    print(doc.page_content)
# Only: "Refunds in 5 business days."

When to compress vs rerank

Gotchas

Compression adds latency and cost. LLM-based compression requires an LLM call per retrieved chunk. For 5 retrieved chunks, that's 5 extra API calls before the main generation. Use a cheap, fast model (Haiku) for compression, not your main generation model.

Compression can drop critical context. Extractive compression might remove sentences that seem irrelevant but are needed for context. Test on edge cases where the answer requires background context that's not directly about the query.

Chunking at indexing time is the upstream fix. If your chunks are noisy, the root cause is often that chunk size is too large. Smaller, more focused chunks at indexing time reduce the need for runtime compression — and reduce latency and cost at query time.

Filter before compress. Always rerank or filter to top-3 before compressing. Compressing 20 chunks is expensive; compressing 3 well-chosen chunks is reasonable.