Metadata-Version: 2.4
Name: rag-ablations
Version: 0.1.0
Summary: Reproducible retrieval benchmark: measures RAG design choices against a stated BM25 baseline
Author: Muhammad Zuhaib Zahid
License: MIT License
        
        Copyright (c) 2026 Muhammad Zuhaib Zahid
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: benchmark,bm25,information-retrieval,rag,retrieval
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: certifi>=2024.2.2
Requires-Dist: numpy>=1.24
Requires-Dist: snowballstemmer>=2.2
Provides-Extra: dense
Requires-Dist: sentence-transformers>=3.0; extra == 'dense'
Provides-Extra: dev
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: service
Requires-Dist: fastapi>=0.110; extra == 'service'
Requires-Dist: uvicorn>=0.29; extra == 'service'
Description-Content-Type: text/markdown

# rag-ablations

[![CI](https://github.com/muhzuhaib/rag-ablations/actions/workflows/ci.yml/badge.svg)](https://github.com/muhzuhaib/rag-ablations/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)

**Most RAG projects claim they improved retrieval. Almost none say what they improved it over.**

This is a retrieval benchmark that always states its baseline. Every design choice (chunking,
sparse vs dense vs hybrid, reranking) is measured against classical BM25 on public corpora with
real relevance judgments, and every number in this README is regenerated by one command.

It runs on a laptop CPU in minutes, with no API keys and nothing to pay for, so anyone reading this
can reproduce the table rather than take it on trust. CI re-runs the benchmark on every push and
fails if a number moves, so the README cannot quietly drift away from the code.

## Is the baseline itself correct?

That question decides whether anything else here means anything, so it is checked against published
figures rather than asserted. The BEIR paper reports Anserini BM25 at **0.665** nDCG@10 on SciFact
and **0.325** on NFCorpus. This independent implementation:

| Corpus | Published Anserini BM25 | This implementation |
|---|---|---|
| SciFact | 0.665 | **0.6862** |
| NFCorpus | 0.325 | **0.3219** |

Within a point on both. The baseline is sound, so the comparisons below are worth reading.

## Results

SciFact: 5,183 documents, 300 test queries, single-threaded CPU.

| System | nDCG@10 | Recall@100 | ms/query |
|---|---|---|---|
| BM25 (stemmed), *baseline* | 0.6862 | 0.9209 | 0.8 |
| BM25 (no stemming) | 0.6663 | 0.8859 | 0.5 |
| Dense (MiniLM-L6) | 0.6451 | 0.9250 | 18.1 |
| **Hybrid (BM25 + dense, RRF)** | **0.7153** | **0.9550** | 35.7 |
| BM25 + cross-encoder rerank | 0.6875 | 0.9209 | 7932.7 |
| Hybrid + cross-encoder rerank | 0.6874 | 0.9550 | 7670.6 |

Three findings, and two of them are negative:

**Dense retrieval lost to BM25:** 0.6451 against 0.6862, at 20× the query cost and five minutes of
indexing against one second. This is not a broken implementation; it is the well-documented BEIR
result that bi-encoders trained on MS MARCO degrade out of domain, and SciFact's scientific claims
are firmly out of domain. Note the split signal: dense had *higher* recall@100 (0.9250) and worse
nDCG@10, meaning it found the relevant documents but ranked them worse.

**Hybrid was the only real win:** 0.7153, beating the baseline by 0.029 nDCG@10 and lifting
recall@100 from 0.9209 to 0.9550. That the fusion beats both of its inputs is the point: the two
retrievers fail on different queries, which is exactly the case RRF exploits, and the recall lift is
evidence they are finding different documents rather than agreeing.

**The cross-encoder reranker was not worth it.** On top of BM25 it moved nDCG@10 by 0.0013, a tie.
On top of hybrid it *lost* 0.028, dragging the best system back down to baseline. And it cost about
**7.9 seconds per query against 36 milliseconds**, roughly 200× slower, because a cross-encoder
scores 100 query-document pairs per search instead of one dot product.

That last result is the one worth dwelling on. Reranking is close to a default recommendation in RAG
write-ups, and here it was neutral at best, harmful on the strongest pipeline, and catastrophic for
latency. The mechanism is visible in the recall column: reranking cannot change recall@100 when it
reranks exactly the top 100, so all it can do is reshuffle, and an MS MARCO cross-encoder
reshuffling out-of-domain scientific claims reshuffled them slightly wrong.

None of this generalises to your corpus. That is the argument for measuring rather than adopting.



## Does chunking help?

Chunking is the RAG design choice most often decided by copying `chunk_size=512, overlap=50` from a
tutorial. Held against a fixed BM25 retriever, on these corpora, it does not help:

SciFact:

| Chunking | nDCG@10 | Recall@100 |
|---|---|---|
| **Whole document (control)** | **0.6862** | **0.9209** |
| 128-word windows, 32 overlap | 0.6794 | 0.9126 |
| 3-sentence windows, stride 2 | 0.6578 | 0.9016 |
| 64-word windows, 16 overlap | 0.6500 | 0.9076 |

NFCorpus:

| Chunking | nDCG@10 | Recall@100 |
|---|---|---|
| **Whole document (control)** | **0.3223** | 0.2464 |
| 128-word windows, 32 overlap | 0.3164 | 0.2413 |
| 3-sentence windows, stride 2 | 0.3085 | 0.2429 |
| 64-word windows, 16 overlap | 0.3076 | 0.2415 |

**Smaller chunks were monotonically worse, and the popular default was worse than doing nothing.**
That is not a general law; it is what these corpora say. Both are built from abstracts and short
articles that already sit near a single topic, so splitting them mostly dilutes term statistics and
splits evidence across passages that then compete with each other. On long, multi-topic documents
the result would likely invert. The point of measuring is that you do not have to guess which case
you are in.

Note that chunking also costs query time: pooling chunk scores back to documents roughly doubled
latency here for no gain.

## Run it

```bash
pip install rag-ablations

python -m rag_ablations.benchmark --dataset scifact --systems sparse     # baseline, no model downloads
python -m rag_ablations.benchmark --dataset scifact --systems chunking   # chunking ablation
python -m rag_ablations.benchmark --dataset scifact --systems all        # adds dense + reranking
```

The corpus downloads itself on first use (2.8 MB) and is cached in `data/`, alongside the
`results/` the run writes, both under whichever directory you run from. No keys, no accounts.
`--systems all` needs the dense extra (`pip install "rag-ablations[dense]"`) and downloads two small
models from HuggingFace.

To work on it rather than run it, clone and `pip install -e ".[dev]"`. In a checkout the same two
directories sit next to `src/`, so `--check` compares against the committed results.

Add `--check` to any command to verify the stored results still reproduce instead of overwriting
them. That is what CI runs.

## The service

```bash
docker compose up
curl "http://localhost:8000/search?q=lithium&k=3"
curl "http://localhost:8000/health"
```

The API builds its retriever through the same factory the benchmark uses, so the published numbers
describe what the service actually does, and `/health` reports the configuration that produced them.
CI builds the image, starts it and issues a real query on every push.

## Design decisions

**Public corpora with relevance judgments, not a private document set.** The alternative, indexing
some documents and eyeballing whether the answers look good, cannot produce a number anyone can
check. BEIR's SciFact and NFCorpus ship human judgments and published baselines, which is what makes
these results falsifiable.

**BM25 as the baseline, written from scratch.** BM25 still beats dense retrieval on many
out-of-domain corpora, so it is a real opponent rather than a straw man. It is implemented here
instead of imported from `rank_bm25` because it is the load-bearing number in the project: `k1` and
`b` are benchmark variables, and a baseline nobody can inspect is a baseline nobody should believe.

**Metrics implemented directly rather than via `pytrec_eval`.** That package needs a C build
toolchain, which breaks "reproducible by anyone on a laptop". nDCG and recall are forty lines, and
[`tests/test_metrics.py`](tests/test_metrics.py) checks them against values computed by hand.

**Two analyzers reported, not one.** Published BM25 baselines use a stemming, stopword-removing
Lucene analyzer. Reporting only the naive tokeniser would have understated the baseline by two
points and quietly flattered everything measured against it; reporting only the stemmed one would
hide how much of the score comes from the analyzer. Both are in the results.

**RRF for hybrid retrieval, not score blending.** BM25 scores are unbounded sums of idf terms while
cosine similarities sit in [-1, 1], so blending them requires normalising two distributions whose
shape changes per query, a tuning knob that quietly becomes a per-dataset fit. RRF uses rank
position only and has a single constant, left at the published 60 rather than tuned against the test
set. Choosing the method with fewer degrees of freedom is what keeps the comparison honest.

**Recall@100 reported alongside nDCG@10.** A reranker can only reorder what the first stage
retrieved, so first-stage recall is a hard ceiling on the whole pipeline. Without it, a reranking
result is uninterpretable, and [one of the tests](tests/test_fusion.py) pins exactly that: a
document the first stage missed cannot be recovered.

**Chunk scores are max-pooled back to documents.** Chunking changes the unit of retrieval, but the
relevance judgments are written against whole documents. Scoring chunks directly would compare
against the wrong ground truth, so every row stays scored against the same qrels.

**Exhaustive cosine search, not an ANN index.** At a few thousand documents, HNSW would add a
dependency and an approximation error to a search that already takes milliseconds. Approximate
indexes belong in a service at a corpus size that justifies them, not in a benchmark whose job is to
isolate the effect of the model.

**Local embeddings, no embedding API.** Partly cost, mainly reproducibility: a hosted endpoint can
change what it returns under a stable model name, and then the table describes a system nobody can
reconstruct. The embedding cache is keyed on the corpus contents, so a chunking change cannot
silently reuse stale vectors.

**`data/` and `results/` are resolved, not assumed.** The package was written inside a checkout,
where those directories sit next to `src/` and the committed results are what `--check` compares
against. Installed from a wheel there is no checkout, and the same relative walk lands in the
interpreter's library directory, so a plain install followed by a benchmark run would download a
corpus into `site-packages` or fail where that is not writable. A checkout is now identified by its
`pyproject.toml`; anywhere else the working directory is used, and `RAG_ABLATIONS_HOME` overrides
both.

**Dense retrieval is an optional install.** `pip install rag-ablations` should not pull 2 GB of
PyTorch on someone who wants the baseline and the metrics.

**The benchmark runs in CI, not just the unit tests.** The tests assert properties (that rare terms
outweigh common ones, that term frequency saturates), so a refactor that quietly changed the
analyzer or the idf formula would pass them all. `--check` re-runs the benchmark over 300 real
queries and fails on drift.

**`certifi` pinned for the corpus download.** Python does not use the Windows trust store and does
not chase AIA links for missing intermediate certificates, so the stdlib default fails on hosts that
`curl` and every browser accept. Found on the first real download, not guessed at.

## Limitations

- **Two corpora, both scientific.** SciFact and NFCorpus were chosen because they are small enough
  to run on a CPU and have published baselines. Conclusions here, especially the chunking result,
  should not be read as general.
- **No hyperparameter search.** `k1`, `b` and `rrf_k` sit at their published defaults. Tuning them
  against the test set would produce better numbers and worse evidence.
- **Single run per system.** These retrievers are deterministic, so there is no variance to report,
  but that also means no confidence intervals on the differences. Small gaps, the 0.0013 between
  BM25 and BM25-plus-reranking for instance, should be read as ties, not as wins.
- **Dense, hybrid and reranking were run on SciFact only.** The reranked configurations take about
  40 minutes on a laptop CPU, so NFCorpus has the sparse and chunking ablations but not the dense
  ones yet.
- **CI reproduces the sparse and chunking results, not the dense ones.** Gating `--systems all`
  would put two model downloads and a 40-minute cross-encoder run on every push. The rows that CI
  cannot re-verify are the rows whose numbers you should trust least.

## License

MIT
