BM25

Why BM25 still matters

Dense retrieval gets all the excitement, but BM25 is the cockroach of information retrieval — it's been around since 1994 and still outperforms dense methods on keyword-heavy queries. If someone searches "Python 3.11 asyncio bug CVE-2023-1234", BM25 finds the exact document in milliseconds. Dense retrieval might find "a blog post about async programming" instead.

For production RAG systems, BM25 is your first baseline and your hybrid-search companion. Build it first; add dense retrieval on top.

The BM25 formula

BM25 scores a document D for query Q by summing over query terms:

score(D,Q) = Σ IDF(qᵢ) × f(qᵢ,D) × (k₁+1)
                              ─────────────────────────────
                              f(qᵢ,D) + k₁×(1-b + b×|D|/avgdl)

• IDF(qᵢ) — inverse document frequency: rare terms score higher
• f(qᵢ,D) — term frequency in document D
• |D|/avgdl — document length normalisation (longer docs are penalised)
• k₁ (default 1.5) — term saturation: repeated terms have diminishing returns
• b (default 0.75) — length normalisation strength

No training, no embeddings, no GPU — just counting and division.

BM25 in Python with rank_bm25

pip install rank-bm25 nltk

from rank_bm25 import BM25Okapi
import nltk
nltk.download("punkt_tab", quiet=True)
from nltk.tokenize import word_tokenize

# Documents
corpus = [
    "Refunds are accepted within 30 days of purchase with original receipt.",
    "Free shipping is available on all orders above fifty dollars.",
    "Customer support is available Monday through Friday, 9 AM to 5 PM.",
    "We accept Visa, Mastercard, American Express, and PayPal.",
]

# Tokenise (lowercase + basic tokenisation)
tokenised = [word_tokenize(doc.lower()) for doc in corpus]

# Build BM25 index
bm25 = BM25Okapi(tokenised)

# Query
query = "return refund receipt"
tokenised_query = word_tokenize(query.lower())
scores = bm25.get_scores(tokenised_query)

# Get top-k results
import numpy as np
top_k_idx = np.argsort(scores)[::-1][:3]
for i in top_k_idx:
    print(f"Score {scores[i]:.3f}: {corpus[i]}")

BM25 in Elasticsearch/OpenSearch

Elasticsearch uses BM25 as its default relevance scorer. A simple search query leverages it automatically:

from elasticsearch import Elasticsearch

es = Elasticsearch("http://localhost:9200")

# Index a document
es.index(index="docs", id="1", document={
    "content": "Refunds accepted within 30 days with original receipt.",
    "category": "policy"
})

# BM25 search (default relevance scorer)
results = es.search(
    index="docs",
    body={
        "query": {
            "match": {
                "content": "return refund receipt"   # BM25 scoring
            }
        }
    }
)
for hit in results["hits"]["hits"]:
    print(f"Score {hit['_score']:.3f}: {hit['_source']['content']}")

Elasticsearch's BM25 is production-grade: it handles multi-field search, language analysers (stemming, stopwords), and horizontal scaling across shards.

When BM25 beats dense retrieval

BM25 consistently outperforms dense retrieval when:

• Exact terms matter — product codes, error messages, CVE numbers, person names
• Domain is highly specialised — medical codes, legal citations, technical specifications where the pre-trained embedding model has little domain knowledge
• Very short queries — 1–2 word queries have insufficient semantic signal for dense retrieval to outperform keyword matching
• Low-resource languages — most dense models were trained on English; BM25 works in any language without training

BM25 as a RAG fallback

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, util
import nltk
from nltk.tokenize import word_tokenize

class HybridRetriever:
    def __init__(self, docs):
        self.docs = docs
        tokenised = [word_tokenize(d.lower()) for d in docs]
        self.bm25 = BM25Okapi(tokenised)
        self.model = SentenceTransformer("all-MiniLM-L6-v2")
        self.doc_embs = self.model.encode(docs, convert_to_tensor=True)

    def retrieve(self, query: str, k: int = 5) -> list[str]:
        # BM25
        bm25_scores = self.bm25.get_scores(word_tokenize(query.lower()))
        # Dense
        q_emb = self.model.encode(query, convert_to_tensor=True)
        dense_scores = util.cos_sim(q_emb, self.doc_embs)[0].cpu().numpy()
        # Normalise and combine (RRF)
        import numpy as np
        combined = (dense_scores / (dense_scores.max() + 1e-9) +
                    bm25_scores / (bm25_scores.max() + 1e-9))
        top_k = np.argsort(combined)[::-1][:k]
        return [self.docs[i] for i in top_k]

Gotchas

BM25 is case and form sensitive. Lowercase and stem your corpus and query consistently. "Refund" and "refunds" are different tokens unless you apply stemming (use NLTK's PorterStemmer or a spaCy pipeline).

rank_bm25 builds in RAM. The in-memory BM25Okapi index is rebuilt every process restart. For persistence, use Elasticsearch/OpenSearch, or pickle the BM25 object (though the pickle may be large for big corpora).

Short documents inflate TF scores. The b parameter penalises shorter documents less — by default, a 10-word document with one match scores comparably to a 1000-word document with five matches. Tune b if your corpus has very variable document lengths.