Metadata-Version: 2.5
Name: lecore-bm25
Version: 0.1.0
Summary: Okapi BM25 + reciprocal rank fusion, with the stoplist/stemming tokenizer that makes it beat stock BM25 out of the box
Project-URL: Homepage, https://github.com/staccDOTsol/lecore-bm25
Project-URL: Upstream, https://github.com/AnOversizedMooseWithSocks/leCore
Author: AnOversizedMooseWithSocks
Maintainer: staccDOTsol
License: MIT License
        
        Copyright (c) 2026 AnOversizedMooseWithSocks
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: beir,bm25,information-retrieval,okapi,rank-fusion,retrieval,rrf,search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.9
Requires-Dist: numpy>=1.20
Description-Content-Type: text/markdown

# lecore-bm25

Okapi BM25 + Reciprocal Rank Fusion, pure NumPy/stdlib, deterministic.

The reason to use this instead of the usual pip BM25 is **the tokenizer ships with it**. That
turns out to be the whole ballgame — see [the numbers](#the-numbers), which are stated with the
decomposition that makes them honest.

```bash
pip install lecore-bm25
```

## Credit where it's due

This is [leCore](https://github.com/AnOversizedMooseWithSocks/leCore)'s
`holographic/semantic_router/holographic_bm25.py`, vendored and packaged. **The algorithm, the
tokenizer, the API and the docstrings are AnOversizedMooseWithSocks'**, MIT licensed, shipped
here with his LICENSE verbatim. The only change is a doc-major postings build that replaces an
`O(vocab x N)` loop that didn't terminate at BEIR-NQ scale; the original is kept beside it as
`_build_postings_vocab_major` and a test asserts the two are bit-identical.

If you want the full library — holographic memory, semantic routing, the rest — go upstream.
This package is just the lexical half, for people who want `pip install` and a good BM25.

## Quickstart

```python
from lecore_bm25 import BM25, tokenize, reciprocal_rank_fusion

docs = [
    "smooth out the bumpy surface of a mesh",
    "denoise a grainy image with a median filter",
    "subdivide a polygon mesh into smaller pieces",
]

bm = BM25(docs)              # k1=1.5, b=0.75 (Robertson defaults)
bm.rank("bumpy surface")     # -> [(0, 4.19...), (2, 0.71...), (1, 0.0)]
bm.scores("bumpy surface")   # -> np.ndarray, one score per doc

# fuse with any other ranker (no score calibration needed)
reciprocal_rank_fusion([[0, 2, 1], [2, 0, 1]], k=60)
```

`tokenize` is the part that matters and it's exported on purpose — stoplist plus light
inflectional and derivational stemming. You can hand it to any other retriever.

## The numbers

BEIR via the `mteb/*` HuggingFace datasets, scored with `pytrec_eval` `ndcg_cut.10` — the same
scorer `mteb` uses underneath — 1000-doc scoring pool, `ignore_identical_ids` on ArguAna.

nDCG@10:

| | SciFact | NFCorpus | ArguAna |
|---|---|---|---|
| **lecore-bm25** | **0.6679** | **0.3185** | **0.4300** |
| pip `rank_bm25`, as its README uses it | 0.5597 | 0.2671 | 0.3448 |
| pip `rank_bm25` + this package's `tokenize` | 0.6664 | 0.3192 | **0.4835** |

**Read the third row before you quote the second.** Against `rank_bm25` with the tokenization its
README actually demonstrates (`doc.lower().split()`, since it ships no tokenizer at all), this
wins by 10.8, 5.1 and 8.5 points. But hand `rank_bm25` this package's `tokenize` and the gap
evaporates. So:

> The scoring math is not better. The tokenizer is the entire advantage.

That's still a real advantage — it's the difference between what you get out of the box and what
you get after you go build a stoplist and a stemmer yourself — but it is a packaging win, not an
algorithmic one, and anyone telling you otherwise is selling something.

### Where this loses: long queries

On ArguAna the third row doesn't just match us, it **beats us by 5.4 points** (0.4835 vs 0.4300).
That is a real limitation and here is the mechanism, so you can decide if it applies to you:

```python
for t in sorted(set(q_terms)):   # lecore-bm25: query terms DEDUPED
for q in query:                  # rank_bm25:   every occurrence counts
```

This implementation drops query-term frequency — a word repeated five times in your query scores
the same as a word appearing once. For keyword-length queries that is invisible (terms rarely
repeat) and it buys reproducibility. On ArguAna, where every "query" is a full argument passage,
it throws away real signal and costs 5.4 points.

**So:** if your queries are short, use this. If your queries are passage-length, use
`rank_bm25`'s scoring with this package's `tokenize` — which is three lines and strictly better:

```python
from rank_bm25 import BM25Okapi
from lecore_bm25 import tokenize
bm = BM25Okapi([tokenize(d) for d in docs])
bm.get_scores(tokenize(query))
```

Two more findings worth recording:

- **The `expand=True` knob is noise.** +0.0026 SciFact, −0.0014 NFCorpus, +0.0008 ArguAna. It is
  off by default and you should leave it off.
- **Nothing here is "holographic."** It's Robertson/Sparck-Jones BM25 with a good tokenizer.

These were independently reproduced from a fresh clone on different hardware by a tester in
Moose's Telegram, matching to four decimals, before being re-run here.

## Reproducing

The bench harness lives in the [supercontext](https://github.com/openzoo/supercontext) bench
campaign (`bm25_vs_pip_bench.py`). It re-runs all three tasks against pip `rank_bm25` and writes
the table above.

## API

- `BM25(docs, k1=1.5, b=0.75)` — `docs` is a list of raw strings
  - `.scores(query, expand=False)` → `np.ndarray` of length N
  - `.rank(query, top=None, expand=False)` → `[(doc_index, score), ...]` descending
- `tokenize(text)` → `list[str]`
- `reciprocal_rank_fusion(ranked_lists, k=60, top=None, weights=None)` → `[(doc, score), ...]`

## License

MIT — Copyright (c) 2026 AnOversizedMooseWithSocks. See `LICENSE`.
