Metadata-Version: 2.4
Name: bm25q
Version: 0.0.1
Summary: Faster, API-compatible BM25 with quality-bounded quantized retrieval.
Home-page: https://github.com/xhluca/bm25s-quantized
Author: Xing Han Lù
Author-email: git@xinghanlu.com
License: MIT
Project-URL: Homepage, https://bm25q.github.io
Project-URL: Source, https://github.com/xhluca/bm25s-quantized
Project-URL: Issues, https://github.com/xhluca/bm25s-quantized/issues
Keywords: bm25 search retrieval sparse quantization information-retrieval
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Provides-Extra: core
Requires-Dist: orjson; extra == "core"
Requires-Dist: tqdm; extra == "core"
Requires-Dist: PyStemmer; extra == "core"
Requires-Dist: numba>=0.62; python_version >= "3.10" and extra == "core"
Requires-Dist: numba; python_version < "3.10" and extra == "core"
Provides-Extra: stem
Requires-Dist: PyStemmer; extra == "stem"
Provides-Extra: hf
Requires-Dist: huggingface_hub; extra == "hf"
Provides-Extra: dev
Requires-Dist: black; extra == "dev"
Provides-Extra: selection
Requires-Dist: jax[cpu]; extra == "selection"
Provides-Extra: indexing
Requires-Dist: scipy; extra == "indexing"
Provides-Extra: evaluation
Requires-Dist: pytrec_eval; extra == "evaluation"
Provides-Extra: mcp
Requires-Dist: mcp; extra == "mcp"
Provides-Extra: cli
Requires-Dist: rich; extra == "cli"
Provides-Extra: full
Requires-Dist: orjson; extra == "full"
Requires-Dist: tqdm; extra == "full"
Requires-Dist: PyStemmer; extra == "full"
Requires-Dist: numba>=0.62; python_version >= "3.10" and extra == "full"
Requires-Dist: numba; python_version < "3.10" and extra == "full"
Requires-Dist: PyStemmer; extra == "full"
Requires-Dist: huggingface_hub; extra == "full"
Requires-Dist: black; extra == "full"
Requires-Dist: jax[cpu]; extra == "full"
Requires-Dist: scipy; extra == "full"
Requires-Dist: pytrec_eval; extra == "full"
Requires-Dist: mcp; extra == "full"
Requires-Dist: rich; extra == "full"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<div align="center">

# bm25q

**Fast BM25 retrieval with quality-bounded quantization.**

[![PyPI](https://img.shields.io/pypi/v/bm25q)](https://pypi.org/project/bm25q/)
[![Python](https://img.shields.io/pypi/pyversions/bm25q)](https://pypi.org/project/bm25q/)
[![License](https://img.shields.io/github/license/xhluca/bm25s-quantized)](LICENSE)

[Website](https://bm25q.github.io) · [PyPI](https://pypi.org/project/bm25q/) · [Source](https://github.com/xhluca/bm25s-quantized)

</div>

`bm25q` is a faster alternative to `bm25s` that keeps the same indexing,
tokenization, retrieval, save/load, and Hugging Face APIs. Exact retrieval is
still the default. Quantized retrieval is opt-in.

The adaptive mode uses one-byte posting impacts and accumulators only when a
per-query upper bound proves that the accumulator cannot overflow. Queries
that do not satisfy that proof automatically use the higher-precision path.

## Install

```bash
pip install "bm25q[core]"
```

The base package only requires NumPy. The recommended `core` extra adds the
stemmer, progress utilities, fast JSON support, and Numba retrieval backend.

## Quick start

```python
import bm25q
import Stemmer

corpus = [
    "a cat likes to purr",
    "a dog loves to play",
    "a fish lives in water",
]

stemmer = Stemmer.Stemmer("english")
corpus_tokens = bm25q.tokenize(corpus, stopwords="en", stemmer=stemmer)

retriever = bm25q.BM25(
    corpus=corpus,
    backend="numba",
    quantize="adaptive",
)
retriever.index(corpus_tokens)

query_tokens = bm25q.tokenize(
    "where does a fish live?",
    stopwords="en",
    stemmer=stemmer,
)
documents, scores = retriever.retrieve(query_tokens, k=2)
```

Existing code only needs an import change:

```python
import bm25q

retriever = bm25q.BM25()
tokens = bm25q.tokenize(["document one", "document two"])
retriever.index(tokens)
```

## Retrieval modes

| Setting | Behavior |
|---|---|
| `quantize=False` | Exact floating-point retrieval; default |
| `quantize=True` | Fast 8-bit impacts with a 16-bit accumulator |
| `quantize="adaptive"` | Quality-bounded one-byte retrieval with automatic higher-precision fallback |

Quantization can slightly reorder documents with nearly identical scores.
Across the full 15-dataset BEIR validation, the largest relative change in
NDCG@10 or Recall@1000 was 0.474%.

## Performance

Single-threaded, full-query local measurements with `k=1000`:

| Dataset | Quantized QPS | Adaptive QPS | Speedup |
|---|---:|---:|---:|
| Natural Questions | 421.5 | **558.2** | **1.32x** |
| MSMARCO | 130.8 | **268.9** | **2.06x** |
| 11-dataset aggregate | 412.8 | **624.4** | **1.51x** |

The paired all-BEIR Kaggle sweep measured 205.77 aggregate QPS for the regular
quantized mode and 243.76 QPS for adaptive mode (1.18x), with all 53,359 qrels
queries and the same benchmark settings for both modes.

## Save and load

```python
retriever.save("index", corpus=corpus)
reloaded = bm25q.BM25.load("index", load_corpus=True)
```

## Optional extras

```bash
pip install "bm25q[hf]"         # Hugging Face Hub integration
pip install "bm25q[cli]"        # terminal interface
pip install "bm25q[evaluation]" # BEIR-style evaluation helpers
pip install "bm25q[full]"       # everything
```

The command-line entry point is `bm25q`:

```bash
bm25q index documents.jsonl -c text -o index
bm25q search -i index "search terms"
```

## License

MIT
