Metadata-Version: 2.5
Name: bilbysearch
Version: 0.10.1
Summary: Article embeddings, vector and BM25 keyword search, and reranking locally, through OpenRouter or with Jev.
Project-URL: Homepage, https://github.com/bilbyai/bilbysearch
Project-URL: Repository, https://github.com/bilbyai/bilbysearch
Author: Samson
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: bm25s>=0.3.11
Requires-Dist: fsspec
Requires-Dist: gcsfs
Requires-Dist: google-cloud-storage
Requires-Dist: jieba>=0.42.1
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: psycopg[binary]>=3.3.6
Requires-Dist: pyarrow
Requires-Dist: requests
Requires-Dist: scikit-learn>=1.9.1
Requires-Dist: sentence-transformers>=5.4
Requires-Dist: tiktoken>=0.14.0
Requires-Dist: torch
Provides-Extra: all
Requires-Dist: bilbystats[llm]>=0.2.25; extra == 'all'
Provides-Extra: embed
Provides-Extra: gcs
Provides-Extra: rerank
Requires-Dist: bilbystats[llm]>=0.2.25; extra == 'rerank'
Description-Content-Type: text/markdown

# bilbysearch

This repo is a work in progress - it is a package for dedicated search code in future search pipelines at Bilby.

**bilbysearch** turns articles into vectors - by default with
[intfloat/multilingual-e5-large-instruct](https://huggingface.co/intfloat/multilingual-e5-large-instruct),
or with any of the 31 other models in its registry, locally or through OpenRouter - and reads
and writes the daily parquet files those vectors live in. It searches those articles by
meaning (vector search) or by keyword (BM25, in English or Chinese), and reranks the
results - with a cross-encoder on this machine, a reranker on OpenRouter, or Jev. Long
articles can be embedded and searched a chunk at a time, so text past a model's context
window is not lost, and a search's ranking can be scored against relevance labels -
recall, precision, nDCG and AP. It also reads the entities database - the people
[entity_profiles](https://github.com/bilbyai/entity_profiles) tracks - into DataFrames.

It exists so each model's contract lives in one place. What a model expects - its
output width, context window, query instruction and document prefix - is recorded once,
in [`embedding_model_config.py`](bilbysearch/embedding/embedding_model_config.py), and every
function takes the model by name and applies that record. Which model production uses,
and pipeline rules such as the `title+body` join, are in
[`production_config.py`](bilbysearch/embedding/production_config.py). Rerankers have a registry of
their own, [`reranker_config.py`](bilbysearch/reranking/reranker_config.py).
So notebooks, batch backfills and retrieval pipelines cannot drift apart.

## Install

```bash
pip install bilbysearch
```

That one command brings embedding, storage, search and reranking: the model stack
(sentence-transformers, torch), `gs://` support (fsspec, gcsfs, google-cloud-storage),
BM25 keyword search (bm25s, and jieba to split Chinese into words), reranking with a
local cross-encoder or through OpenRouter, queries on the entities database (psycopg,
whose wheel bundles the Postgres client library), and the retrieval metrics
(scikit-learn, which sentence-transformers brings anyway). torch is the heavy part, at
around 2 GB once CUDA wheels are involved.

Only reranking with Jev is optional, because it brings bilbystats and its LLM client
libraries. The extra is called `rerank` because it predates the other rerankers, which
need nothing extra:

```bash
pip install "bilbysearch[rerank]"
```

`[all]` installs everything. The `[embed]` and `[gcs]` extras from 0.1.x are still
accepted, so an existing `"bilbysearch[embed,gcs]"` dependency keeps working, but they no
longer add anything.

## Tutorials

[`tutorials/`](tutorials/README.md) walks through the package in notebooks, each runnable
top to bottom on a few articles built in the notebook. Start with
[`getting_started.ipynb`](tutorials/getting_started.ipynb), which embeds a day of
articles, stores it, searches it by meaning and by keyword, and reranks what the searches
find. The [index](tutorials/README.md) lists the rest in reading order, with what each
needs to run: most need nothing set up, and those that call OpenRouter say how many
requests they make. Each section below links to the tutorial for its part.

## Usage

```python
import bilbysearch

# Embed a frame of articles. Rows with no text get a null vector rather than a
# meaningless one, and three provenance columns record how the vectors were made.
articles = bilbysearch.read_parquet("daily_2026-01-31.parquet")
embedded = bilbysearch.embed_dataframe(articles)
bilbysearch.write_parquet(embedded, "daily_2026-01-31_embedding.parquet")
```

Queries and documents are embedded by **different functions on purpose**: the
production model expects an instruction prefix on queries and none on documents, and
skipping it measurably degrades retrieval. `embed_query` adds the query instruction and
`embed_texts` the document prefix, so neither can be forgotten or put on the wrong side.

```python
query = bilbysearch.embed_query("Chinese monetary policy easing")
passages = bilbysearch.embed_texts(["The PBoC cut the reserve requirement ratio."])
similarity = query @ passages[0]  # vectors are L2-normalised, so this is cosine
```

The model is loaded on first use and kept loaded - the two most recently used models
stay in memory - so embedding a frame and then a query loads it once.
`clear_embedder_cache()` releases them. Every function also takes `device` (`"cuda"`,
`"mps"` or `"cpu"`; the default lets sentence-transformers choose).

Progress - which model is loading, how many rows are embedded, each OpenRouter batch - is
logged at `INFO`, through loggers named after the modules (`bilbysearch.embedding.embed`,
`bilbysearch.storage`, ...). The package configures no logging of its own: warnings, such
as an OpenRouter request being retried, reach stderr with no setup, and to see the
progress too, ask for it:

```python
import logging

logging.basicConfig()
logging.getLogger("bilbysearch").setLevel(logging.INFO)
```

### Choosing a model

Every function takes the model's name as `model_name`, defaulting to the production
model. The name is all you pass: its width, context window and prefixes come from its
entry in the registry.

```python
bilbysearch.list_local_embedding_models(language="zh")  # "en", "zh" or "multilingual"
bilbysearch.get_model_spec("BAAI/bge-m3")                # what the registry records

embedded = bilbysearch.embed_dataframe(articles, model_name="intfloat/e5-large-v2")
query = bilbysearch.embed_query("PBoC easing", "intfloat/e5-large-v2")  # gets "query: "
```

That dataframe's articles were embedded with e5-large-v2's `"passage: "` prefix, and its
`embedding_model` column says `intfloat/e5-large-v2`: the one name picks the model and
labels the output, so the two cannot disagree. Embed queries with the same model as the
documents; vectors from two models are not comparable.

Mistakes fail before anything is loaded or sent:

- **An unknown name** is refused, with the nearest known names suggested
  (`"baai/bge-m3"` -> did you mean `'BAAI/bge-m3'`?).
- **A text that already starts with its model's prefix** is refused, e.g.
  `"passage: ..."` for e5-large-v2, which would otherwise be embedded as
  `"passage: passage: ..."`. Pass texts without prefixes.
- **A loaded model that disagrees with its registry width** is refused, so its vectors
  never reach a vector store sized from the registry.

**To switch production to another model**, change `MODEL_NAME` in
[`production_config.py`](bilbysearch/embedding/production_config.py) to a local model from the
registry; its width, context window and prefixes follow.
[`tests/embedding/test_production_config.py`](tests/embedding/test_production_config.py) pins the production
values, so update it in the same change. Existing vectors must be re-embedded, and a
vector store of a different width recreated.

**To add an embedding model**, add its entry to
[`embedding_model_config.py`](bilbysearch/embedding/embedding_model_config.py): its output width
(measured from real output, not read from its card), context window, query instruction
and document prefix (copied character for character from its card), language, whether
it needs `trust_remote_code`, and where it runs.

[`local_small_model_embeddings.ipynb`](tutorials/embedding/local_small_model_embeddings.ipynb)
runs every embedding step with a 90 MB model, and shows how to try another.

### Through OpenRouter

The same steps can run on OpenRouter's servers instead of this machine, for the models
the registry records as hosted there (`list_openrouter_embedding_models()`; the
production model is not one of them). Set `OPENROUTER_API_KEY`, then pass the model's
registry name - the id OpenRouter uses and the model's prefixes come from its entry:

```python
model_name = "intfloat/multilingual-e5-large"

embedded = bilbysearch.embed_dataframe_openrouter(articles, model_name=model_name)
passages = bilbysearch.embed_texts_openrouter(["The PBoC cut the RRR."], model_name)
query = bilbysearch.embed_query_openrouter("PBoC easing", model_name)
```

Locally, a text longer than the model's context window is cut short: e5-large reads
only its first 512 tokens. OpenRouter would receive the whole text, so where the
registry names the model's tokenizer (e5-large and the Perplexity models), each text is
cut to the same window with that tokenizer before it is sent, and OpenRouter embeds
exactly what the local model reads. OpenAI's models refuse a text over their 8,191-token
window outright - one such text fails its whole batch - so their texts are cut with
tiktoken, OpenAI's own tokenizer, before they are sent. Gemini's tokenizer is not
public, so its texts are sent whole, and OpenRouter cuts each to Gemini's 2,048-token
window itself, keeping the start. It bills the whole text, though, so a long article
costs more than the part that is embedded.

Every text is also capped at 256 KiB (`MAX_INPUT_BYTES` in
[`embed_openrouter.py`](bilbysearch/embedding/embed_openrouter.py)), between whole
characters. A text that is mostly whitespace can fit inside a model's window and still be
megabytes long, and the provider's web server refuses such a request (HTTP 413) before
any model reads it. A genuine article cut to its window stays well under the cap.

To count the tokens OpenRouter bills, pass a dict as `usage`. Each response's counts -
`prompt_tokens`, `total_tokens`, and any other count OpenRouter reports - are added to
it as the response arrives, so one dict totals a whole run, and a run that fails partway
still counts the batches already answered:

```python
usage = {}
embedded = bilbysearch.embed_dataframe_openrouter(articles, model_name=model_name, usage=usage)
query = bilbysearch.embed_query_openrouter("PBoC easing", model_name, usage=usage)
usage["prompt_tokens"]  # every token billed for the frame and the query
```

Before mixing OpenRouter and local vectors in one index, check they match:

```python
report = bilbysearch.compare_embeddings(local_vectors, openrouter_vectors)
report["all_match"]  # True if every pair has cosine similarity >= 0.99
```

[`check_openrouter_matches_local.py`](examples/embedding/embed_openrouter/check_openrouter_matches_local.py)
runs this check for multilingual-e5-large.

#### Threading

`embed_texts_openrouter` and `embed_dataframe_openrouter` send texts in batches of
`batch_size` (default 32) per request, and use threads to keep up to
`max_concurrent_requests` (default 8) requests in flight at once. Vectors come back in
input order. Lower `max_concurrent_requests` if OpenRouter rate-limits you; `1` sends
one request at a time.

```python
embedded = bilbysearch.embed_dataframe_openrouter(
    articles, model_name=model_name, max_concurrent_requests=4
)
```

For async code, `embed_texts_openrouter_async` and `embed_query_openrouter_async` take
the same arguments and can be awaited without blocking the event loop:

```python
query = await bilbysearch.embed_query_openrouter_async("PBoC easing", model_name)
```

[`embed_openrouter.ipynb`](tutorials/embedding/embed_openrouter.ipynb) runs each of these,
and checks OpenRouter's vectors against the same model run locally.

### Upgrading from 0.8

0.9 groups the modules by job - `embedding/`, `vector_search/` and `keyword_search/`,
beside `reranking/` and `entities/` - so a module's path says which part of the
pipeline it belongs to. Everything imported from `bilbysearch` itself, such as
`from bilbysearch import embed_dataframe`, is unchanged. A moved module imported by its
old path fails at the import, with `ModuleNotFoundError`:

| 0.8 | 0.9 |
|---|---|
| `bilbysearch.embed` | `bilbysearch.embedding.embed` |
| `bilbysearch.embed_openrouter` | `bilbysearch.embedding.embed_openrouter` |
| `bilbysearch.embedding_model_config` | `bilbysearch.embedding.embedding_model_config` |
| `bilbysearch.production_config` | `bilbysearch.embedding.production_config` |
| `bilbysearch.chunking` | `bilbysearch.embedding.chunking` |
| `bilbysearch.compare` | `bilbysearch.embedding.compare` |
| `bilbysearch.bm25_search` | `bilbysearch.keyword_search.bm25_search` |
| `bilbysearch.vector_search` | a package at the same path: `vector_search`, `CHUNK_SCORINGS` and `DEFAULT_TOP_K` import from it as before, and the module's other names, such as `stack_chunk_vectors`, are in `bilbysearch.vector_search.vector_search` |
| `OPENROUTER_API_KEY_ENV`, `RETRYABLE_STATUSES` or `openrouter_error` from `bilbysearch.embed_openrouter` | from `bilbysearch.openrouter_client`, or `OPENROUTER_API_KEY_ENV` from `bilbysearch` |
| `LANGUAGES` from `bilbysearch.embedding_model_config` | from `bilbysearch` |
| `bilbysearch.model_config`, deprecated since 0.7 | removed: `bilbysearch.embedding.embedding_model_config`, or `from bilbysearch import MODELS` |

### Upgrading from 0.6

0.7 adds local and OpenRouter rerankers, and with them a reranker registry, so the
embedding registry's module is renamed to say which models it holds. Code written for
0.6 keeps running, unless it passed `rerank_jev` an argument that could never be right:

| 0.6 | 0.7 |
|---|---|
| `from bilbysearch.model_config import MODELS` | `from bilbysearch.embedding_model_config import MODELS`, or `from bilbysearch import MODELS`. The old import kept working, with a `DeprecationWarning`, until 0.9 removed it. |
| `rerank_jev(q, df, top_n=-1)` quietly kept all but the last row | raises `ValueError`, as do a `top_n` or `max_chars` below 1 and a `min_relevance` outside 0 to 1 - before any request is made |

### Upgrading from 0.2

0.3 takes models by name. Code written for 0.2 fails with an error saying what to pass
instead, rather than running with the wrong settings:

| 0.2 | 0.3 |
|---|---|
| `model = load_embedder(name)`, then `embed_texts(texts, model)` | `embed_texts(texts, name)` |
| `embed_query(q, model, instruction="query: ")` | `embed_query(q, name)` |
| `embed_dataframe(df, model=model, model_name=name)` | `embed_dataframe(df, model_name=name)` |
| `embed_texts(["passage: " + t for t in texts], model)` | `embed_texts(texts, name)` |
| `embed_query_openrouter(q, name, instruction="query: ")` | `embed_query_openrouter(q, name)` |
| `vector_search(q, df, model, instruction=...)` | `vector_search(q, df, name)` |
| `from bilbysearch.config import ...` | `from bilbysearch.production_config import ...`, or `from bilbysearch import ...` |
| any model id | a name in `bilbysearch.MODELS` |

## Searching

`vector_search` scores every article in an embedded frame against a query and returns the
closest, best first, with a `cosine_similarity` column. To search stored days, read
them into one frame first:

```python
uris = [bilbysearch.parquet_uri(prefix, date, "_embedding") for date in dates]
embedded = pd.concat([bilbysearch.read_parquet(uri) for uri in uris], ignore_index=True)
candidates = bilbysearch.vector_search("Chinese monetary policy easing", embedded, top_k=100)
```

The query is embedded with `model_name` - the production model unless you say
otherwise - and it has to be the model that made the article vectors. Frames from
`embed_dataframe` record that model in their `embedding_model` column, so a search
with a different one, or over a frame that mixes days embedded by two models, is
refused rather than returning a meaningless ranking.

It is brute force: every vector is loaded and compared in memory. That is exact and
fine for weeks of articles; a corpus too big to load at once needs a vector store.

[`vector_search.ipynb`](tutorials/vector_search/vector_search.ipynb) searches embedded
days, and [`storage.ipynb`](tutorials/storage.ipynb) covers the daily files they are read
from, and a backfill that embeds each day once.

### Keyword search with BM25

`vector_search` matches meanings; `bm25_search` matches words. It finds exact names,
codes and figures that an embedding can blur, and misses the paraphrases an embedding
catches. It needs no embeddings, so it runs on the raw daily files. Build an index once,
then search it as often as needed:

```python
day = bilbysearch.read_parquet("daily_2026-01-31.parquet")
index = bilbysearch.build_bm25_index(day, language="zh")  # title+body, split with jieba
hits = bilbysearch.bm25_search("央行 降准", index)         # milliseconds per search
hits[["title", "bm25_score"]]
```

**Choose the language and the columns together.** A query only matches words in its own
language, and each language is split into words differently. In the daily files
`title+body` is mostly Chinese and `title_en+body_en` is English:

```python
en_index = bilbysearch.build_bm25_index(day, language="en", text_col="title_en+body_en")
hits = bilbysearch.bm25_search("reserve requirement cut", en_index)
```

- `"en"` splits text the way bm25s does by default - lowercased words of two or more
  letters or digits, English stopwords removed - so results match the team's English
  BM25 baseline.
- `"zh"` splits Chinese into words with [jieba](https://github.com/fxsjy/jieba)'s search
  mode, which also emits the shorter words inside long ones: 中国人民银行 gives 中国,
  人民, 银行 and the whole name, so a query for any of them finds it. Full-width letters
  and digits (ＧＤＰ, ４.８％) read as GDP and 4.8%, and Latin terms in Chinese text
  (PBoC, 5G) are kept.

`language` has no default, and an English index refuses a query with Chinese in it
rather than silently matching nothing.

**What comes back.** Up to `top_k` (default 100) articles that contain at least one of
the query's words, best first, with a `bm25_score` column - ready to rerank.
Articles with none of the words are left out, so fewer than `top_k` can come back. A
score depends on the whole index as well as the article - how rare each word is across
it, how long its articles are on average - so it ranks articles within one search. It
is not comparable across indexes, and is not a relevance cutoff.

**Tuning.** `build_bm25_index` takes BM25's two parameters. `k1` (default 1.5) sets how
much repeating a word raises a score: higher lets each repeat keep adding, lower makes a
word count about the same however often it appears. `b` (default 0.75) sets how much
long articles are discounted, from 0 (not at all) to 1 (in proportion to their length).
bm25s works out each word's contribution to each article's score when the index is
built, which is what makes searches fast, so `k1` and `b` are fixed for an index: build
another to try other values.

**Cost.** Building reads every article: 2-3 s on a laptop for a day's Chinese
`title+body` (around 1,300 articles), and 0.3 s for its English translations. Each
search after that takes milliseconds. The index lives in memory.

**Known limits of `"zh"`:**

- jieba's dictionary is simplified Chinese, so traditional text is split poorly.
- A name the dictionary does not know is split into pieces - 宁德时代 gives 宁德 and 时代 -
  which match more loosely than the whole name would.
- Search mode trades some precision for recall: 人民币 (the yuan) also gives 人民, which
  is in 中国人民银行 (the People's Bank of China), so a query about the yuan also finds
  People's Bank articles, lower down.

[`ex_bm25_search.py`](examples/keyword_search/bm25_search/ex_bm25_search.py) runs these searches on a
few articles, and [`bm25_search.ipynb`](tutorials/keyword_search/bm25_search.ipynb) shows
what each language is split into, tunes `k1` and `b`, and sets BM25 beside
`vector_search` on a stock code and a paraphrase.

## Chunking long articles

A model reads at most its context window - 512 tokens for the production model - and
silently drops the rest, so an article's one vector describes only its opening. In the
English 50-query experiment in semantic-search-tests, 63% of articles were longer than
that, and a passage further in cannot be found by its meaning, however well it matches.

With `chunking=True`, each article is split into overlapping chunks that the model reads
whole, and every chunk is embedded. The frame keeps one row per article:

```python
embedded = bilbysearch.embed_dataframe(articles, chunking=True)
embedded["chunk_embeddings"]  # a list of vectors per article, one per chunk
embedded["chunk_spans"]       # each chunk's [start, end] in the article's title+body

hits = bilbysearch.vector_search("semiconductor export controls", embedded, chunking=True)
hits[["title", "cosine_similarity", "best_chunk_span"]]
```

`vector_search` scores every chunk and each article by its best one, so an article comes
back once however many of its chunks match, `top_k` counts articles, and
`best_chunk_span` says which passage matched. The hits go to a reranker as they are.

To score each article by the average of its chunks instead, pass `chunk_scoring="mean"`:

```python
hits = bilbysearch.vector_search(query, embedded, chunking=True, chunk_scoring="mean")
```

The mean favours articles about the query throughout over one that matches in a single
passage. In [`ex_vector_search.py`](examples/vector_search/vector_search/ex_vector_search.py), the
sports round-up found by its last paragraph drops from second to third. `"max"`, the
default (`DEFAULT_CHUNK_SCORING`), is what semantic-search-tests benchmarked; `"mean"`
was not tested there.

**How articles are cut.** A chunk is a window of the model's own tokens. `chunk_size`
defaults to as many as fit beside the model's special tokens and document prefix - 510
for the production model - and `chunk_overlap` to an eighth of that, 64, so text near a
cut is read with context on both sides:

```python
embedded = bilbysearch.embed_dataframe(articles, chunking=True, chunk_size=256, chunk_overlap=32)
```

- A chunk is the article's own characters, never tokens decoded back into text, which
  can differ from the original (some tokenizers put spaces between Chinese characters).
- Every chunk is tokenized again, prefix and all, and shortened if it would run over.
  That check is needed: a chunk cut from the middle of a text can take a token or two
  more on its own than it did in place, and in a sample of mixed Chinese and English
  articles, a third of chunks cut at exactly 510 tokens ran over 512.
- An article that fits is one chunk, and gets exactly the vector it gets unchunked.
- `chunk_size` and `chunk_overlap` are refused without `chunking=True`, rather than
  ignored, and a `chunk_size` larger than the model reads is refused.
- A long-context model's default chunk is so large that most articles are a single
  chunk - bge-m3 reads 8,192 tokens, Qwen3 32,768. Pass a `chunk_size` to chunk them.

**Storing and searching.** Pass the same `chunking` everywhere the frame goes:

```python
stored = bilbysearch.select_output_columns(embedded, id_col="uuid", chunking=True)
bilbysearch.write_parquet(stored, "daily_2026-01-31_embedding.parquet")
```

The stored file holds the id, the chunk vectors, their spans and the provenance - not
the chunks' text, which stays in the daily file:
`bilbysearch.select_text_series(day)[row][start:end]` is a chunk. A frame stored or
searched with the wrong `chunking` is refused, saying which kind of vectors it holds.
`DEFAULT_CHUNKING` in [`production_config.py`](bilbysearch/embedding/production_config.py) is the
default for all four functions, so one change switches embedding, storing and searching
together - and means re-embedding every stored day.

**Through OpenRouter**, `embed_dataframe_openrouter` takes the same arguments. Chunks are
cut with the HuggingFace tokenizer the registry names, so only e5-large and the
Perplexity models can be chunked there; chunking OpenAI's models, which tiktoken cuts
instead, or Gemini, whose tokenizer is not public, is refused. e5-large is cut into the
same chunks locally and on OpenRouter.

**When it helps.** Chunking is off in production because, for the production model, it
found no more: over 848 queries in each language, recall@100 moved by less than 0.001
(English 0.9918 to 0.9921, Chinese 0.9952 to 0.9947; the Chinese difference is not
significant), while embedding took twice as long (112 minutes against 51 for the Chinese
run). It pays off where the passage that matters sits past the model's window -
[`ex_vector_search.py`](examples/vector_search/vector_search/ex_vector_search.py) finds a sports
round-up by its last paragraph only when chunked - and for weaker Chinese models: on 50
Chinese queries, recall@100 rose from 0.80 to 0.88 for zpoint and from 0.88 to 0.90 for
text2vec, though it fell for multilingual-e5-large, from 0.95 to 0.92.

[`chunking.ipynb`](tutorials/embedding/chunking.ipynb) walks through all of this, on an
English and a Chinese article whose most relevant passage the model would otherwise never
read.

## Reranking

Cosine similarity finds articles that are *about the same things* as the query, and BM25
finds articles that *use the same words*. A reranker reads the query with each candidate
and scores whether the candidate is actually relevant. Pass one the top 50 to 100
candidates from `vector_search` or `bm25_search`:

```python
query = "China semiconductor export controls"
candidates = bilbysearch.vector_search(query, embedded)

reranked = bilbysearch.rerank_local(query, candidates, model_name="BAAI/bge-reranker-v2-m3")
reranked = bilbysearch.rerank_openrouter(query, candidates, model_name="cohere/rerank-v3.5")
reranked = bilbysearch.rerank_jev(query, candidates)
```

All three take the same arguments - `text_col`, `top_n`, `min_relevance`, and `max_chars`
(default 2,000, since relevance is decided by an article's opening) - and return a copy
of the frame sorted by relevance, highest first. Ties keep the incoming order, and the
original index is kept. A `top_n` or `max_chars` below 1, or a `min_relevance` outside 0
to 1, is refused before anything is loaded or sent.

| | Runs | Needs | Adds |
|---|---|---|---|
| `rerank_local` | on this machine | the base install | `rerank_relevance`, `rerank_model` |
| `rerank_openrouter` | on OpenRouter | `OPENROUTER_API_KEY` | `rerank_relevance`, `rerank_model` |
| `rerank_jev` | TypeSafe's Jev, through OpenRouter | the `[rerank]` extra and `OPENROUTER_API_KEY` | `jev_relevance`, `jev_confidence`, `jev_probabilities` |

All three are built on `add_relevance_columns`, which holds those shared rules: give it a
function that scores a list of texts to rerank another way.

### Local and OpenRouter rerankers

`rerank_local` and `rerank_openrouter` take the reranker by its name in the reranker
registry, as the embedding functions take models. There is no default, as no reranker
has been chosen for production yet.

```python
bilbysearch.list_local_reranker_models(language="multilingual")
# ['BAAI/bge-reranker-v2-m3', 'Qwen/Qwen3-Reranker-0.6B']
bilbysearch.list_openrouter_reranker_models()
bilbysearch.get_reranker_spec("cohere/rerank-v3.5")  # what the registry records
```

| Reranker | Runs | Language | Reads (tokens) |
|---|---|---|---|
| `BAAI/bge-reranker-v2-m3` | locally | multilingual | 8,192 |
| `Qwen/Qwen3-Reranker-0.6B` | locally | multilingual | 40,960 |
| `cross-encoder/ms-marco-MiniLM-L6-v2` | locally (90 MB, fast) | English | 512 |
| `mixedbread-ai/mxbai-rerank-large-v1` | locally | English | 512 |
| `cohere/rerank-v3.5` | on OpenRouter | multilingual | 4,096 |
| `cohere/rerank-4-fast`, `cohere/rerank-4-pro` | on OpenRouter | multilingual | 32,768 |
| `voyageai/rerank-2.5`, `voyageai/rerank-2.5-lite` | on OpenRouter | multilingual | 32,000 |
| `qwen/qwen3-reranker-8b` | on OpenRouter | multilingual | 40,960 |

Both add the same two columns whichever model ran: `rerank_relevance`, the model's score
from 0 to 1, and `rerank_model`, the model that made it. Code that reads the scores need
not know which reranker made them, and a saved frame says whose scores it holds. A
second rerank of the same frame replaces both columns together, so to compare two
rerankers, run each on the same candidates and join them:

```python
bge = bilbysearch.rerank_local(query, candidates, model_name="BAAI/bge-reranker-v2-m3")
small = bilbysearch.rerank_local(query, candidates, model_name="cross-encoder/ms-marco-MiniLM-L6-v2")
both = bge.join(small[["rerank_relevance"]], rsuffix="_small")
```

**The scores are each model's own.** They rank articles for one query and one model. They
are not calibrated probabilities, as Jev's are, so a `min_relevance` tuned for one model
means nothing for another. In [`ex_rerank_local.py`](examples/reranking/rerank/ex_rerank_local.py),
bge-reranker-v2-m3 scores a Chinese article on the query 0.93, while the English-only
ms-marco-MiniLM, which cannot read it, scores it 0.00 and the English one 0.999. Use a
multilingual reranker on a mixed-language corpus.

**Locally**, each score is the sigmoid of the model's raw output, applied by
`rerank_local` rather than left to the model, since models disagree about it. The model
is loaded on first use and the two most recently used stay loaded: `load_reranker` loads
one ahead of time, and `clear_reranker_cache()` releases them. A row with no text gets
no score and sorts last, and is never shown to the model.

**On OpenRouter**, set `OPENROUTER_API_KEY`. Candidates go in batches of `batch_size`
(default 100, one Cohere search) with up to `max_concurrent_requests` (default 4) in
flight, and rate limits and server errors are retried, as for the OpenRouter embeddings.
Every candidate is scored - `top_n` cuts the frame, not the request - and a response that
misses a document, or scores one off the 0-1 scale, raises rather than misplacing scores.
Cohere charges per search, and Voyage and Qwen per token, which `max_chars` keeps down.
[`check_openrouter_rerankers.py`](examples/reranking/rerank_openrouter/check_openrouter_rerankers.py) runs
every hosted reranker once, for under a cent.

**To add a reranker**, add its entry to
[`reranker_config.py`](bilbysearch/reranking/reranker_config.py): how many tokens it reads
(for a local model, measured from the loaded CrossEncoder), its language, and where it
runs. For one on OpenRouter, run `check_openrouter_rerankers.py` too: OpenRouter can list
a model that no provider serves, as it listed `qwen/qwen3-reranker-0.6b` in October 2026,
so Qwen3-Reranker-0.6B runs only locally here. Two models that bring their own code -
`Alibaba-NLP/gte-multilingual-reranker-base` and
`jinaai/jina-reranker-v2-base-multilingual` - were considered and left out, because
neither runs under transformers 5.

[`rerank_local.ipynb`](tutorials/reranking/rerank_local.ipynb) and
[`rerank_openrouter.ipynb`](tutorials/reranking/rerank_openrouter.ipynb) rerank the same
candidates with several models, side by side, and `rerank_local.ipynb` writes a reranker
of its own with `add_relevance_columns`.

### Jev

`rerank_jev` asks [TypeSafe's Jev](https://developers.cloudflare.com/ai/models/typesafe/jev/)
whether each candidate is relevant, and re-sorts by its calibrated probability. It needs
the `[rerank]` extra (see Install). The call goes through `bilbystats.batch_jev_api`,
routed through OpenRouter by default, so it needs an `OPENROUTER_API_KEY` in the
environment. Pass `provider="typesafe"` or `provider="vercel"` to route elsewhere, each
with its own key.

```python
reranked = bilbysearch.rerank_jev(query, candidates, min_relevance=0.5)
reranked[["title", "jev_relevance", "jev_confidence", "jev_probabilities"]]
```

`jev_relevance` is the probability that the article is relevant, `jev_confidence` is
Jev's confidence in that answer and `jev_probabilities` is the full yes/no dict.
Because the probability is absolute, `min_relevance` works as a cutoff as well as a
rank. [`jev_reranking.ipynb`](tutorials/reranking/jev_reranking.ipynb) runs vector search
then Jev end to end, and sets Jev beside a local reranker.

## Evaluating retrieval

Given relevance labels for a query's pool of articles, `evaluate` scores how well a
search ranked them: recall, precision, nDCG and AP at each cutoff.

```python
scores = np.array([0.91, 0.85, 0.80, 0.42, 0.40, 0.12])  # one per article in the pool
labels = np.array([1, 0, 1, 0, 2, 0])  # 0 not relevant, 1 relevant, 2 more relevant

metrics = bilbysearch.evaluate(scores, labels, ks=[2, 5])
# recall@2 0.333, precision@2 0.5, nDCG@2 0.380, MAP@2 0.333,
# recall@5 1.0,   precision@5 0.6, nDCG@5 0.726, MAP@5 0.756   (rounded)
```

`recall_at_k`, `precision_at_k`, `ndcg_at_k` and `ap_at_k` compute one metric at one
cutoff, and `evaluate` with no `ks` reports the cutoffs in `DEFAULT_KS`: 5, 10, 20 and 50.

- **Pass the whole pool.** Recall and AP divide by every relevant article in `labels`, so
  a pool cut down to a search's top hits loses the relevant articles the search missed,
  and overstates both.
- **Graded labels.** Recall, precision and AP count an article as relevant when its label
  is above 0, each article once. nDCG uses the label itself as the gain, so ranking a
  label-2 article above a label-1 one scores higher. Labels must be integers: float labels
  are refused, since they usually mean the scores and labels were passed the wrong way
  round.
- **Ties rank in pool order.** numpy's default sort orders equal scores differently from
  one machine to another, and BM25 scores most of a pool 0, so recall, precision and AP
  rank tied articles in the order they were passed, the earlier first. nDCG averages the
  gain over each group of tied articles instead, as scikit-learn computes it.
- **AP@k** divides by every relevant article in the pool, not just those in the top k, as
  TREC's MAP@k does, so it falls short of 1 when the pool holds more relevant articles
  than k. The `MAP@k` key holds one query's AP: average it over queries for MAP.
- **precision@k** divides by k even when the pool is smaller than k.
- A pool with **no relevant article** scores 0 on every metric.

[`ex_evaluate.py`](examples/evaluation/metrics/ex_evaluate.py) builds a frame of results
with one row per query and averages it into MAP; the other scripts in its folder show
each metric on its own.

## Querying the entities database

`entity_profiles.entities` holds the people
[entity_profiles](https://github.com/bilbyai/entity_profiles) tracks - heads of state,
ministers, central bank governors - one row per entity, in a Postgres database on
Railway. `query_entities_database` runs a SQL query on it and returns the rows as a
DataFrame.

It connects to the URL in `ENTITIES_POSTGRES_URL`. Use the Railway Postgres service's
`DATABASE_PUBLIC_URL`: its `DATABASE_URL` resolves only inside Railway.

```bash
export ENTITIES_POSTGRES_URL=postgresql://...
```

```python
chinese_entities = bilbysearch.query_entities_database(
    """
    SELECT name_en, name_zh, standardised_role
    FROM entity_profiles.entities
    WHERE country = 'People''s Republic of China'
    ORDER BY name_en
    """
)
```

- **Read-only.** Each call opens a read-only connection and sends one statement, so
  Postgres refuses writes - and SQL holding more than one statement - rather than
  applying them. This guards against mistakes, not intent: the URL's credentials can
  still write from any other client, so a read-only Postgres role is the real lock.
- **Values from variables** go in `params`, not into the SQL with an f-string, so a value
  can never be read as SQL - `help(bilbysearch.query_entities_database)` shows how.
  A value typed into the SQL, as above, doubles any apostrophe: `'People''s'`.
- **jsonb columns** - roles, aliases, sources - come back as Python lists and dicts.

[`tutorials/entities/entities_table.ipynb`](tutorials/entities/entities_table.ipynb)
describes every column and lists the countries the table holds.

## Development

```bash
uv sync --all-extras
uv run pytest
uv run ruff check . && uv run ruff format .
```

With conda instead of uv, install from `requirements.txt`. It pins every package - the
`[rerank]` extra and the dev tools included - at its version in `uv.lock`, and installs
this checkout as an editable package. Run it from the repo root:

```bash
conda create -n bilbysearch python=3.12
conda activate bilbysearch
pip install -r requirements.txt
pytest
```

`requirements.txt` is exported from `uv.lock`, never edited by hand: after changing a
dependency, run the `uv export` command in its header. A test fails until you do.

The tests use stub models and a faked OpenRouter, so the suite runs in under twenty
seconds, never downloads a real model and never makes an API call. A few chunking tests
check the cuts against e5's real tokenizers, and run only where those are already
cached. The entities database tests start a throwaway local Postgres instead of reaching
Railway, and are skipped where Postgres's `initdb` and `pg_ctl` are not on PATH.

The package is grouped by job: `embedding/` makes the vectors, `vector_search/` and
`keyword_search/` find candidates, `reranking/` re-sorts them and `entities/` reads the
entities database, with the modules they share - `model_lookup.py`,
`openrouter_client.py` and `storage.py` - directly in `bilbysearch/`. `tests/` and
`examples/` mirror that layout: the tests for `bilbysearch/embedding/embed.py` are in
`tests/embedding/test_embed.py`, and the examples of its functions are in
`examples/embedding/embed/`, one `ex_<function>.py` each. `tutorials/` follows it too,
with each notebook beside the part it teaches, and
[`tests/test_tutorials.py`](tests/test_tutorials.py) fails on any file a tutorial - or
this README - names that is not there. Dependencies point one way: the
searches and rerankers build on `embedding/`, which builds on the shared modules.
[`tests/test_dependency_direction.py`](tests/test_dependency_direction.py) records which
part may import which, and fails on an import that points the other way, or on a new
module until its place there is decided.

## Licence

MIT — see [LICENSE](LICENSE).
