BM25
A probabilistic lexical ranking function that scores documents by exact term frequency, providing the keyword precision that vector search alone cannot.
1 / What BM25 Does
BM25 (Best Matching 25) is a probabilistic ranking function that scores documents based on term frequency, inverse document frequency, and document length normalization. Unlike Semantic Search, which operates on meaning through Vector Embeddings, BM25 operates on exact lexical matches — the literal words that appear in the query and the document.
2 / Why It Matters in Hybrid Retrieval
In the Hybrid RAG pipeline I built for Phoenix, BM25 provided the keyword precision that dense vector retrieval could not. When a user searched for a specific technical term — an API name, a configuration parameter, a protocol identifier — BM25 reliably surfaced documents containing that exact term, even when the embedding model did not capture the specificity of the match.
The implementation used the `rank_bm25` Python library to score documents against query terms. The BM25 candidate set ran in parallel with the pgvector semantic retrieval, and the two result sets were merged through Reciprocal Rank Fusion before Reranking.
3 / Tuning Parameters
BM25's behavior is controlled by two parameters: `k1` (term frequency saturation) and `b` (document length normalization). Higher `k1` values increase the importance of term frequency, while `b` controls how much longer documents are penalized. For the technical documentation in Phoenix, moderate values prevented long documents from being unfairly penalized while maintaining the importance of exact term matches.
4 / Limitations
BM25 cannot understand meaning. 'Machine learning models' and 'neural networks' are semantically related but share no terms — BM25 would treat them as completely unrelated queries. This is precisely why hybrid retrieval combines BM25 with vector search: each strategy covers the other's blind spot.