The standard RAG retrieval stack has become a ritual: embed the documents, keep a BM25 index too, fuse the two lists with reciprocal rank fusion, then run a cross-encoder reranker over the top of the fused list. Each step comes with a blog post saying it helps.
I wanted to know how much each step helps, with a confidence interval, on a dataset with real relevance labels. So I built the four pieces myself, in about 100 lines of numpy, and ran them on SciFact from BeIR: 5,183 scientific abstracts and 300 claims, each labelled with the abstracts that support or refute it.
The pieces
BM25 is lexical: a document scores on how many query terms it contains, weighted by how rare each term is across the corpus and discounted for document length. Stemmed, stopwords removed, k1 = 0.9 and b = 0.4.
Dense retrieval embeds query and documents with bge-small-en-v1.5 and ranks by cosine similarity. The whole thing is one matrix multiply.
Reciprocal rank fusion is the part people get mystical about, and it’s four lines:
def rrf(rankings, k=60):
fused = Counter()
for ranking in rankings:
for rank, doc in enumerate(ranking, 1):
fused[doc] += 1 / (k + rank)
return [doc for doc, _ in fused.most_common()]That’s the formula from Cormack, Clarke and Büttcher at SIGIR 2009, with their k of 60. It never looks at a score, which is the point: a BM25 score of 14.2 and a cosine of 0.83 have nothing to say to each other, but “third on one list and fifth on the other” does. For comparison I also ran convex fusion, which min-max normalises both score lists and averages them, which is what most vector databases do when you ask for hybrid.
Reranking takes the top 20 from a list and rescores each query-document pair with a cross-encoder, ms-marco-MiniLM-L-6-v2, which reads both texts together and can attend across them.
Every comparison below is paired: the same 300 queries under both systems, with a bootstrap over queries for the 95% interval. I’ve argued elsewhere that this is the only honest way to compare two runs, and this benchmark was built to practise it.
Results
| System | nDCG@10 | Recall@100 | MRR | Latency per query |
|---|---|---|---|---|
| BM25 | 0.680 | 0.922 | 0.652 | 2.1 ms |
| Dense (bge-small) | 0.720 | 0.953 | 0.688 | 1.2 ms |
| RRF fusion | 0.734 | 0.965 | 0.706 | ~3 ms |
| Convex fusion | 0.743 | 0.962 | 0.717 | ~3 ms |
| Dense + rerank | 0.702 | 0.953 | 0.672 | 1,437 ms |
| RRF + rerank | 0.706 | 0.965 | 0.676 | 1,437 ms |
And the paired comparisons, which are the part that matters:
| Change | ΔnDCG@10 | 95% CI | Queries better / worse / tied |
|---|---|---|---|
| BM25 → RRF fusion | +0.053 | +0.031 to +0.076 | 82 / 21 / 197 |
| Dense → RRF fusion | +0.013 | −0.010 to +0.037 | 57 / 38 / 205 |
| Convex fusion → RRF fusion | −0.009 | −0.025 to +0.006 | 30 / 33 / 237 |
| Dense → dense + rerank | −0.018 | −0.044 to +0.008 | 44 / 55 / 201 |
| RRF fusion → RRF + rerank | −0.028 | −0.054 to −0.002 | 42 / 60 / 198 |
Fusion helps a lot, or a little, depending on where you start
If your system is BM25 today, adding a dense retriever and fusing is worth 5.3 points, and the interval is nowhere near zero. If your system is already a decent dense retriever, adding BM25 is worth 1.3 points and the interval includes zero. Same technique, same data; the value depends on what it’s being added to.
Where the gain lives is also worth seeing. Recall@100 went from 0.953 to 0.965: fusion put relevant documents into the candidate set for about 1 in 80 queries where dense retrieval alone had missed them. That’s the real job of a second retriever, catching the exact-term matches (a drug name, an error code, a gene symbol) that embeddings smooth over. If you want a quick test of whether your queries need it, look for identifiers in them.
RRF and convex fusion tie
Convex fusion edged RRF by 0.9 points with an interval that includes zero and 237 of 300 queries unchanged. I’d call that a tie, and a tie goes to the method with no parameters to tune. RRF has one constant that the original authors set once and never changed; convex fusion has a weight and a normalisation scheme, and the min-max step is brittle when one list’s scores have a long tail. Use RRF unless you have a measurement that says otherwise.
The reranker made it worse
This is the row I didn’t expect. Reranking the top 20 cost 1.8 points over dense retrieval and 2.8 over fusion, and the second interval excludes zero. It also cost about 1.4 seconds per query on CPU, against 1 to 3 milliseconds for everything else.
Two things are going on. First, a reranker can only reorder what it’s given. Recall@100 is identical with and without it, because the candidate set didn’t change; it can’t rescue a document retrieval missed. Second, this reranker was trained on MS MARCO, which is web search questions over web passages. SciFact is scientific claims over abstracts. A 22-million-parameter cross-encoder with the wrong training distribution lost to a 33-million-parameter bi-encoder with a better one. “Cross-encoders are more accurate” is true in-distribution and on the leaderboards; it’s not a law.
I’m not saying rerankers don’t work. Larger ones, and ones tuned to your domain, usually do. I’m saying that on this corpus the default recipe’s final step was a 700× latency cost for a worse ranking, and nobody would have noticed without a labelled set, because the results still look plausible.
Most queries don’t change at all
Look at the tied column. Between 182 and 237 of 300 queries produced the identical top-10 under both systems in every comparison. Decisions about hybrid search are decided by the 20 to 40% of queries where the systems disagree, which is why per-query win and loss counts tell you more than an average, and why a change can look like a win on the dashboard and lose on more queries than it wins (reranking: 44 better, 55 worse).
What I’d do
- Start from a strong dense retriever and measure before adding BM25. Expect a small gain concentrated in identifier-heavy queries. If your queries have none, you may not need it.
- Fuse with RRF, k = 60. Nothing to tune, nothing to normalise.
- Treat a reranker as a hypothesis. Build a labelled set of your own queries first (50 is enough to catch a regression this size), and only keep the reranker if it wins with the interval clear of zero. Keep the paired bootstrap in CI.
- Chase recall before precision. Fusion’s gain was mostly recall. If the right document isn’t in the top 100, no amount of reordering helps, and chunking decides that before any of this runs.
Limits
One dataset. I planned to run NFCorpus and FiQA too, and the NFCorpus run did not finish on the 8 GB laptop I had, so these are SciFact numbers and SciFact is a particular kind of corpus: short, dense scientific prose with a lot of exact terminology. BM25 parameters weren’t tuned. Both models are small; a larger reranker or one trained on scientific text would likely change the last two rows. Reranking depth was 20.
Code, tests (including a hand-checked RRF case and nDCG against a worked example) and the full results: ajinb/hybrid-retrieval-bench. python bench.py --datasets scifact reproduces everything above; it downloads the dataset and both models on first run. For the basics, start with what RAG is and vector index trade-offs.
Comments