Hybrid Search + Rerank: Fix Embeddings' Blind Spot for Exact Matches

Search for an exact order number in a RAG system built purely on embeddings, and you'll often get nothing useful back. Not because the order number isn't in your corpus — because embeddings were never designed to nail exact-token matches in the first place.

The problem: embeddings and keywords fail differently

Dense embedding search is excellent at capturing meaning. Ask "how do I get a refund" and it can retrieve a chunk about "return policy and reimbursement" even though the words barely overlap — that's the whole point of semantic search. But that same strength is a weakness for queries where the exact string matters: order numbers, SKUs, error codes, product model numbers. Two very different order numbers can end up embedded close together in vector space because the surrounding text is similar, even though — token for token — they have nothing in common.

Keyword search (classically BM25, a scoring function built on term frequency) has the opposite profile: it's built to reward exact term overlap. It nails order numbers and error codes but misses paraphrases — search "how do I get my money back" against a corpus that only says "refund," and pure keyword search comes up empty.

Neither retrieval method covers the other's blind spot. Running only one of them means you're structurally guaranteed to miss a category of queries.

The fix: run both, fuse the results, then rerank

1. Hybrid retrieval. Run BM25 keyword search and dense embedding search against the same query, in parallel, each producing its own ranked list of candidates. Merge the two ranked lists with Reciprocal Rank Fusion (RRF) — a fusion method that combines rankings based on each result's position in each list, rather than trying to normalize and compare two different scoring scales directly.

2. Reranking. Take the fused, merged list — not your whole index, just the top candidates from fusion (a common range is the top 50-ish) — and rerank them with a cross-encoder. Unlike the embedding model used for initial retrieval, which encodes the query and each document independently, a cross-encoder looks at the query and a candidate document together in a single pass. That joint attention makes it dramatically better at judging true relevance, but also much more expensive per comparison — which is exactly why you only run it on a short list of candidates, not the whole corpus.

# 1. retrieve from both indexes
bm25_hits = bm25_index.search(query, k=50)
dense_hits = vector_store.similarity_search(query, k=50)

# 2. fuse rankings (reciprocal rank fusion)
fused = reciprocal_rank_fusion([bm25_hits, dense_hits])

# 3. rerank the fused top candidates with a cross-encoder
top_candidates = fused[:50]
reranked = cross_encoder.rerank(query, top_candidates)

Why this is worth the extra moving part

This isn't a theoretical improvement — it shows up directly in retrieval quality benchmarks. In one documented walkthrough, adding a reranking pass on top of hybrid retrieval pushed NDCG@10 (a standard ranking-quality metric — how well the most relevant results land near the top of the list, not just whether they're present somewhere) from roughly 38 up to 47. In a separate live tool-call trace, enabling reranking was observed raising a query's similarity score to 0.47. That's a meaningfully better set of top results reaching the model, from the same underlying corpus and the same initial retrieval — the difference is entirely in fusion and rerank.

Keep reranking small and precise

The cross-encoder step is the expensive one — it's doing real joint inference on every query-document pair it's handed, not a cheap vector comparison. The pattern that keeps this practical:

• Retrieve broadly and cheaply first (BM25 + embeddings, both fast at scale).
• Fuse down to a manageable candidate set.
• Only rerank that short list — tens of candidates, not thousands.

Reranking your entire index on every query defeats the purpose of having a fast first-stage retriever at all.

Power tricks

• Don't skip the fusion step and just concatenate both lists. Reciprocal rank fusion specifically accounts for a document appearing in both lists (or only one), which naive concatenation or dedup does not.
• Cross-encoder latency adds up. If you're reranking on every request in a latency-sensitive path, benchmark the added time — it's real, and it's why the candidate set needs to stay small.
• Neither BM25 nor embeddings is "the old way." They cover different failure modes; hybrid isn't a stepping stone to pure-embedding search, it's the intended architecture for corpora with both exact-match and paraphrase-style queries.

Resources

• Pinecone — Hybrid search explained
• Sentence-Transformers — retrieve & rerank with cross-encoders
• BM25 (Okapi) — background on the ranking function