Metadata-Version: 2.5
Name: lecore-bm25
Version: 0.2.1
Summary: Okapi BM25 + reciprocal rank fusion, with the stoplist/stemming tokenizer that makes it beat stock BM25 out of the box
Project-URL: Homepage, https://github.com/staccDOTsol/lecore-bm25
Project-URL: Upstream, https://github.com/AnOversizedMooseWithSocks/leCore
Author: AnOversizedMooseWithSocks
Maintainer: staccDOTsol
License: MIT License
        
        Copyright (c) 2026 AnOversizedMooseWithSocks
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: beir,bm25,information-retrieval,okapi,rank-fusion,retrieval,rrf,search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.9
Requires-Dist: numpy>=1.20
Description-Content-Type: text/markdown

# lecore-bm25

Okapi BM25 + Reciprocal Rank Fusion, pure NumPy/stdlib, deterministic.

The reason to use this instead of the usual pip BM25 is **the tokenizer ships with it**. That
turns out to be the whole ballgame — see [the numbers](#the-numbers), which are stated with the
decomposition that makes them honest.

```bash
pip install lecore-bm25
```

## Credit where it's due

This is [leCore](https://github.com/AnOversizedMooseWithSocks/leCore)'s
`holographic/semantic_router/holographic_bm25.py`, vendored and packaged. **The algorithm, the
tokenizer, the API and the docstrings are AnOversizedMooseWithSocks'**, MIT licensed, shipped
here with his LICENSE verbatim. The only change is a doc-major postings build that replaces an
`O(vocab x N)` loop that didn't terminate at BEIR-NQ scale; the original is kept beside it as
`_build_postings_vocab_major` and a test asserts the two are bit-identical.

If you want the full library — holographic memory, semantic routing, the rest — go upstream.
This package is just the lexical half, for people who want `pip install` and a good BM25.

## Quickstart

```python
from lecore_bm25 import BM25, tokenize, reciprocal_rank_fusion

docs = [
    "smooth out the bumpy surface of a mesh",
    "denoise a grainy image with a median filter",
    "subdivide a polygon mesh into smaller pieces",
]

bm = BM25(docs)              # k1=1.5, b=0.75 (Robertson defaults)
bm.rank("bumpy surface")     # -> [(0, 4.19...), (2, 0.71...), (1, 0.0)]
bm.scores("bumpy surface")   # -> np.ndarray, one score per doc

# fuse with any other ranker (no score calibration needed)
reciprocal_rank_fusion([[0, 2, 1], [2, 0, 1]], k=60)
```

`tokenize` is the part that matters and it's exported on purpose — stoplist plus light
inflectional and derivational stemming. You can hand it to any other retriever.

## The numbers

BEIR via the `mteb/*` HuggingFace datasets, scored with `pytrec_eval` `ndcg_cut.10` — the same
scorer `mteb` uses underneath — 1000-doc scoring pool, `ignore_identical_ids` on ArguAna.

nDCG@10:

| | SciFact | NFCorpus | ArguAna |
|---|---|---|---|
| **lecore-bm25** (0.2.0) | **0.6689** | **0.3179** | **0.4867** |
| lecore-bm25 0.1.0 (query terms deduped) | 0.6679 | 0.3185 | 0.4300 |
| pip `rank_bm25`, as its README uses it | 0.5597 | 0.2671 | 0.3448 |
| pip `rank_bm25` + this package's `tokenize` | 0.6664 | 0.3192 | 0.4835 |
| pip `bm25s`, its documented defaults | 0.6617 | 0.3064 | 0.4839 |
| pip `bm25s` + this package's `tokenize` | 0.6689 | 0.3175 | 0.4862 |

**Read rows four through six before you quote row three.** Against `rank_bm25` with the
tokenization its README actually demonstrates (`doc.lower().split()`, since it ships no tokenizer
at all), this wins by 10.9, 5.1 and 14.2 points. That number is real, and it is also the least
impressive thing you could measure, because it is a comparison against the weakest baseline
available. Two things collapse it:

- Hand `rank_bm25` this package's `tokenize` and the gap evaporates.
- Compare against **`bm25s`**, which is maintained and whose documented default tokenization
  already lowercases and drops English stopwords, and the margin is **+0.7, +1.2 and +0.3
  points**. On ArguAna, `bm25s` out of the box *beat* our 0.1.0 outright — that was the
  query-term-frequency bug below, and it took fixing it to pull level.

So:

> The scoring math is not better. The tokenizer is the entire advantage — and against a library
> that already tokenizes sensibly, that advantage is worth about a point.

That's still a real advantage over the most-installed option, and it's the difference between what
you get out of the box and what you get after you go build a stoplist and a stemmer yourself. But
it is a packaging win, not an algorithmic one, and anyone telling you otherwise is selling
something. If you are already using `bm25s` and happy, there is no numerical reason to switch.

One comparison we deliberately do **not** make: the published BEIR Elasticsearch BM25 baselines
(SciFact 0.620, NFCorpus 0.297, ArguAna 0.441). Those are full-corpus; every number in the table
above uses a 1000-document scoring pool, which is an easier problem. Putting them in the same
table would flatter us dishonestly.

### Where this used to lose: long queries (fixed in 0.2.0)

Through 0.1.0 this section documented a defeat: on ArguAna, `rank_bm25` with our own tokenizer
**beat us by 5.4 points** (0.4835 vs 0.4300). The mechanism was one line —

```python
for t in sorted(set(q_terms)):   # 0.1.0: query terms DEDUPED
for q in query:                  # rank_bm25: every occurrence counts
```

— and 0.2.0 fixes it by counting query terms instead of deduping them, so a term repeated q times
contributes q x its weight, which is what BM25's query-side term frequency has always meant.

We measured the fix across seven BEIR tasks before shipping it, because a change that helps one
task and quietly costs the other six is not a fix:

| task | mean query tokens | repeat rate | 0.1.0 | 0.2.0 | delta |
|---|---:|---:|---:|---:|---:|
| NFCorpus | 2.7 | 0.003 | 0.3185 | 0.3179 | −0.0006 |
| Touche2020 | 4.2 | 0.006 | 0.3428 | 0.3428 | +0.0000 |
| TREC-COVID | 7.9 | 0.008 | 0.6176 | 0.6174 | −0.0002 |
| SCIDOCS | 7.9 | 0.016 | 0.1587 | 0.1569 | −0.0018 |
| FiQA | 7.3 | 0.019 | 0.2422 | 0.2414 | −0.0009 |
| SciFact | 9.5 | 0.028 | 0.6679 | 0.6689 | +0.0011 |
| **ArguAna** | **121.6** | **0.230** | 0.4300 | **0.4867** | **+0.0566** |

The effect is entirely explained by how often query terms repeat. Six of seven BEIR tasks have
keyword-length queries that repeat almost nothing, so deduping was invisible there — every delta is
under 0.002, which is noise. ArguAna's "queries" are whole argument passages, and there it was
throwing away real signal.

Be careful what you conclude from that correlation: it looks near-perfect (r = 0.998 against query
length) but it is carried by a single point. Drop ArguAna and it collapses to r = 0.155. The
honest reading is a bound, not a law — harmless up to ~9.5 tokens / 0.028 repeat rate, worth 5.7
points at 121.6 tokens / 0.230. Nothing we measured says where in between it starts to matter.

With the fix, this now edges `rank_bm25`-with-our-tokenizer on ArguAna too (0.4867 vs 0.4835), so
the earlier advice to switch libraries for passage-length queries no longer applies.

Two more findings worth recording:

- **The `expand=True` knob is noise.** +0.0026 SciFact, −0.0014 NFCorpus, +0.0008 ArguAna. It is
  off by default and you should leave it off.
- **Nothing here is "holographic."** It's Robertson/Sparck-Jones BM25 with a good tokenizer.

These were independently reproduced from a fresh clone on different hardware by a tester in
Moose's Telegram, matching to four decimals, before being re-run here.

## Reproducing

The bench harness lives in the [supercontext](https://github.com/openzoo/supercontext) bench
campaign. `bm25_vs_pip_bench.py` re-runs the three tasks against pip `rank_bm25`;
`tokenizer_tax_bench.py` adds `bm25s`, scikit-learn TF-IDF and Whoosh, each in both its own
default tokenization and this package's, which is where the "+0.7 against `bm25s`" number above
comes from. Both write the tables above and gate on reproducing previously published values before
reporting anything new.

## API

- `BM25(docs, k1=1.5, b=0.75)` — `docs` is a list of raw strings
  - `.scores(query, expand=False)` → `np.ndarray` of length N
  - `.rank(query, top=None, expand=False)` → `[(doc_index, score), ...]` descending
- `tokenize(text)` → `list[str]`
- `reciprocal_rank_fusion(ranked_lists, k=60, top=None, weights=None)` → `[(doc, score), ...]`

## License

MIT — Copyright (c) 2026 AnOversizedMooseWithSocks. See `LICENSE`.
