Hybrid RAG
Combining dense vector retrieval with sparse keyword matching for retrieval systems that understand both meaning and exact terminology.
1 / The Problem with Single-Strategy Retrieval
Pure semantic search — embedding queries and documents into vector space and finding nearest neighbors — works well when the user's intent aligns with the meaning captured by the embedding model. But it misses a critical category of queries: exact keyword matches. A search for 'BM25 algorithm' in a pure vector system might retrieve documents about 'ranking algorithms' or 'information retrieval scoring' while missing the document that literally contains the exact phrase.
Conversely, pure keyword search — matching terms directly — misses semantic equivalences. 'Document relevance scoring' and 'ranking' describe the same concept but share no terms. Neither strategy alone is sufficient for robust retrieval.
2 / The Hybrid Architecture
The hybrid approach I implemented in Phoenix ran two retrieval strategies in parallel: dense Semantic Search using Vector Embeddings stored in pgvector, and sparse lexical retrieval using BM25. Each strategy returned its own ranked candidate set with its own scoring scale — cosine similarity for vectors (0 to 1) and BM25 scores (unbounded positive values).
The engineering challenge was merging these fundamentally different score scales into a single ranked list. Reciprocal Rank Fusion (RRF) solved this by converting absolute scores into rank-based scores, making the fusion method agnostic to the underlying score distributions.
3 / Reranking as the Final Stage
After fusion, the top candidates passed through a Reranking stage using a Cross-Encoder model. Unlike bi-encoder embeddings (which encode query and document independently), the Cross-Encoder evaluated query-document pairs jointly, producing more accurate relevance scores at the cost of higher computational overhead. Capping the top-k candidates sent to the reranker kept latency manageable.
4 / Query Rewriting
Vague user queries like 'how does search work' produced poor retrieval results with both strategies. Query rewriting — using an LLM to expand or rephrase the query before retrieval — resolved ambiguity before the retrieval pipeline executed. This preprocessing step improved both semantic and keyword retrieval quality without modifying the retrieval architecture itself.
5 / What I Learned
The hybrid approach delivered measurably better retrieval quality than either strategy alone. The key engineering insight: retrieval pipeline design is about composing complementary strategies with explicit fusion, not choosing a single 'best' algorithm. Each stage — embedding, keyword matching, fusion, reranking — addresses a different failure mode of the previous stage.