Multi-Query Retrieval

The single-query blind spot

You ask "How do I cancel my subscription?" and the most relevant document says "To terminate your account, visit Account Settings." Standard dense retrieval embeds your query as a single vector and looks for nearby documents — but "cancel subscription" and "terminate account" may not be close enough in embedding space to score highly.

This is the vocabulary mismatch problem: your phrasing differs from the document's phrasing, and a single query embedding can only cover one "view" of the semantic space.

Multi-query retrieval solves this by generating 3–5 paraphrased versions of the original query, retrieving documents for each, and merging the results. Each paraphrase covers a different corner of the semantic space, dramatically improving recall.

How multi-query retrieval works

1. Expand: send the original query to an LLM with a prompt asking for N diverse paraphrases ("Generate 4 different phrasings of this question").
2. Retrieve: run a standard retrieval (dense, BM25, or hybrid) for each paraphrase independently. Get top-k results for each.
3. Merge and deduplicate: combine all result sets. If the same document appears in multiple result lists, keep it once (Reciprocal Rank Fusion handles ranking).
4. Rerank optionally: apply a cross-encoder reranker on the merged set to get the final top-N before passing to the LLM.

The key insight: documents that consistently appear across multiple differently-worded retrievals are very likely to be genuinely relevant — multi-query acts as a confidence boost.

Implementation from scratch

import anthropic
from sentence_transformers import SentenceTransformer
import numpy as np

client = anthropic.Anthropic()
model = SentenceTransformer("BAAI/bge-large-en-v1.5")

def generate_query_variants(query: str, n: int = 4) -> list[str]:
    response = client.messages.create(
        model="claude-3-5-haiku-20241022",
        max_tokens=300,
        messages=[{"role": "user", "content": f'''Generate {n} different phrasings of the following question.
Each phrasing should approach the topic from a different angle.
Return ONLY the {n} questions, one per line, no numbering or explanation.

Original question: {query}'''}]
    )
    variants = [q.strip() for q in response.content[0].text.strip().split('\n') if q.strip()]
    return [query] + variants[:n]   # include original + n variants

def multi_query_retrieve(query: str, corpus: list[dict], k: int = 5) -> list[dict]:
    # Generate query variants
    variants = generate_query_variants(query, n=4)
    print(f"Query variants: {variants}")

    # Embed corpus once (offline in practice)
    texts = [d["text"] for d in corpus]
    prefix = "Represent this sentence for searching relevant passages: "
    corpus_embs = model.encode(texts, normalize_embeddings=True)

    # Retrieve for each variant, track doc appearances
    doc_scores: dict[str, list[float]] = {}
    for variant in variants:
        q_emb = model.encode(prefix + variant, normalize_embeddings=True)
        scores = corpus_embs @ q_emb
        top_k_idx = np.argsort(scores)[::-1][:k]
        for rank, idx in enumerate(top_k_idx):
            doc_id = corpus[idx]["id"]
            doc_scores.setdefault(doc_id, []).append(1 / (60 + rank + 1))  # RRF

    # Merge by RRF score
    merged = sorted(doc_scores.items(), key=lambda x: sum(x[1]), reverse=True)
    id_to_doc = {d["id"]: d for d in corpus}
    return [id_to_doc[doc_id] for doc_id, _ in merged[:k]]

# Test
corpus = [
    {"id": "1", "text": "To cancel your subscription, go to Account Settings > Billing > Cancel Plan."},
    {"id": "2", "text": "Terminate your account by visiting the subscription management portal."},
    {"id": "3", "text": "Free shipping applies to orders over fifty dollars."},
    {"id": "4", "text": "Your subscription renews automatically on the billing date unless cancelled."},
]
results = multi_query_retrieve("How do I stop my subscription?", corpus)
for r in results[:3]:
    print(r["text"][:60])

LangChain MultiQueryRetriever

from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain_anthropic import ChatAnthropic
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain.schema import Document
import logging

# Set up logging to see the generated queries
logging.basicConfig()
logging.getLogger("langchain.retrievers.multi_query").setLevel(logging.INFO)

# Build the base vector store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
docs = [
    Document(page_content="To cancel your subscription, go to Account Settings."),
    Document(page_content="Terminate your account via the subscription portal."),
    Document(page_content="Your plan auto-renews unless you cancel beforehand."),
    Document(page_content="Free shipping on all orders above $50."),
]
vectorstore = Chroma.from_documents(docs, embeddings)
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# Wrap with MultiQueryRetriever — uses LLM to generate variants
llm = ChatAnthropic(model="claude-3-5-haiku-20241022")
multi_retriever = MultiQueryRetriever.from_llm(
    retriever=base_retriever,
    llm=llm,
    include_original=True   # always include original query
)

# Retrieve — generates ~3 variants automatically, merges results
results = multi_retriever.invoke("How do I stop paying for the service?")
for doc in results:
    print(doc.page_content)
# Logs show the generated queries; results include docs from all variants

Query diversity strategies

The quality of multi-query retrieval depends on generating diverse variants. Default LLM paraphrasing works, but targeted strategies work better:

DIVERSITY_PROMPT = '''Generate 4 alternative search queries for the original question below.
Use these strategies:
1. Rephrase with synonyms (e.g. "cancel" -> "terminate", "subscription" -> "plan")
2. Ask from a different perspective (e.g. "how to..." -> "what are the steps to...")
3. Use more technical or formal language
4. Use simpler, everyday language

Original: {query}

Return only the 4 alternative queries, one per line.'''

You can also generate queries that target specific document types:

TARGETED_PROMPT = '''Given the question, generate:
1. A version for finding step-by-step instructions
2. A version for finding policy documents
3. A version for finding FAQ answers
4. A version with technical terminology

Original: {query}
'''

Cost vs recall tradeoff

Note that all query variants can be generated in a single LLM call. The extra latency comes from running the retriever N times, not N LLM calls. Use asyncio.gather to parallelise retrieval across variants:

import asyncio

async def retrieve_variant(variant: str) -> list:
    # Your async retrieval function
    return await async_vector_search(variant, k=10)

async def multi_query_async(variants: list[str]) -> list:
    results = await asyncio.gather(*[retrieve_variant(v) for v in variants])
    # Merge results with RRF
    return merge_with_rrf(results)

Gotchas

Generated queries can be off-target. LLMs occasionally produce variants that drift from the original intent. Use a lower temperature (0.0–0.3) for query generation and include the original query in the result set unconditionally.

Deduplication is essential. Multiple variants often retrieve the same documents. Without deduplication, you'll pass the same chunk to the LLM multiple times, wasting context tokens. Use document IDs or content hashing to deduplicate.

Combining with HyDE is powerful. Multi-query generates diverse question phrasings; HyDE generates document-style hypothetical answers. Combining both (generate 3 HyDE documents + 3 query variants = 6 retrieval passes) can push recall above 95% on most benchmarks — at the cost of 7 LLM calls per query.

Don't use multi-query for simple keyword lookups. If the user types an exact product code or error message, generating paraphrases introduces noise. Apply multi-query selectively for natural-language questions, not for exact-match lookups.