Metadata-Version: 2.4
Name: jev-reranker
Version: 0.1.1
Summary: Multilingual Jev reranking with listwise, pointwise, and pairwise scoring
License-Expression: MIT
License-File: LICENSE
Requires-Dist: httpx>=0.28,<1
Requires-Dist: python-dotenv>=1,<2
Requires-Dist: huggingface-hub>=0.30,<2 ; extra == 'all'
Requires-Dist: pyarrow>=20,<24 ; extra == 'all'
Requires-Dist: sentence-transformers>=5,<6 ; extra == 'all'
Requires-Dist: tokenizers>=0.21,<1 ; extra == 'all'
Requires-Dist: sentence-transformers>=5,<6 ; extra == 'sentence-transformers'
Requires-Dist: huggingface-hub>=0.30,<2 ; extra == 'tokenizer'
Requires-Dist: tokenizers>=0.21,<1 ; extra == 'tokenizer'
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/hotchpotch/jev-reranker
Project-URL: Repository, https://github.com/hotchpotch/jev-reranker
Project-URL: Issues, https://github.com/hotchpotch/jev-reranker/issues
Provides-Extra: all
Provides-Extra: sentence-transformers
Provides-Extra: tokenizer
Description-Content-Type: text/markdown

# jev-reranker: Relevance Filtering & Reranking for RAG

[![CI](https://github.com/hotchpotch/jev-reranker/actions/workflows/ci.yml/badge.svg)](https://github.com/hotchpotch/jev-reranker/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/jev-reranker.svg)](https://pypi.org/project/jev-reranker/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

jev-reranker is a Python library for reranking search results and filtering
retrieved documents with TypeSafe.AI's Jev. It provides prompts for both tasks
and handles concurrent requests, splitting long candidate lists, and retries.

For a walkthrough with examples, read
[Introducing jev-reranker: Reranking and Relevance Filtering for RAG](https://huggingface.co/blog/hotchpotch/introducing-jev-reranker).

Search results can match a question without helping answer it. Passing every
match to an LLM adds input tokens and potentially distracting context.
`relevance_rerank()` scores documents for their usefulness as evidence, sorts
them, and removes those below a configurable threshold. If nothing passes,
your application can try another search or stop before generation.

Use it after retrieval and before assembling context for RAG. Use `rerank()`
when you want to reorder candidates without filtering by default. Both accept
a query string and a list of document strings; scoring runs through the Jev API.
The base package needs no local model or GPU.

## Highlights

- Relevance filtering scores documents for their contribution to an answer,
  including partial answers and facts needed for multi-hop reasoning.
- A configurable threshold determines which documents to keep. Results are sorted
  by score; an empty list means no candidate passed the threshold.
- Prompts are Python dictionaries. You can change the instructions and criteria
  to describe what counts as useful evidence in your application.
- Long candidate lists are split automatically. Document limits and split budgets
  can use character counts, a tokenizer, or a custom length function.
- Sync and async methods share request handling, concurrency limits, retries,
  and automatic HTTP cleanup.
- Optional details record all scores, including excluded documents, along with
  the prompts, model, input lengths, API usage, and request history.

## Getting started

Requires Python 3.11+ and a TypeSafe API key. Jev API usage is billed by the service.

```sh
uv add jev-reranker
# Or: pip install jev-reranker
```

Replace `YOUR-TYPESAFE-API-KEY...` in the examples with your TypeSafe API key.
You can also omit `api_key` and set `TYPESAFE_API_KEY` in the environment or `.env`.
Keep real API keys out of version control.
Save either example below as `example.py` and run `uv run python example.py`.

### Filter for useful evidence

```python
from jev_reranker import JevReranker

query = "How long do I have to return an online order to ACME Shop?"
documents = [
    "ACME Shop accepts online returns within 30 days of delivery.",
    "For ACME Shop online orders, submit your return request within 30 days of receiving the item.",
    "ACME Shop in-store purchases can be returned within 14 days of purchase.",
    "ACME Shop products come with a one-year repair warranty.",
    "FooBar Shop accepts online returns within 60 days of delivery.",
]

reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...")
evidence = reranker.relevance_rerank(query, documents, threshold=0.2)

for item in evidence["results"]:
    print(f"{item['score']:.2f}  {item['text']}")
```

Example output (scores can vary by model and input context):

```text
0.98  ACME Shop accepts online returns within 30 days of delivery.
0.97  For ACME Shop online orders, submit your return request within 30 days of receiving the item.
```

Only the two passages about ACME Shop online returns remain. The in-store return,
repair warranty, and FooBar Shop passages fall below the 0.2 threshold and are omitted.
This example uses the default listwise mode; it does not limit the output with
`top_k`.

Results are sorted by descending score. Each result also includes
`document_index`, its position in the original input, so you can retrieve URLs or
other metadata from your search results. If nothing passes the threshold,
`evidence["results"]` is an empty list: your application can try another search or abstain.

Lower thresholds retain more documents; higher thresholds are more selective.
The default keeps scores of 0.2 or higher. Test the threshold on your own queries
to see whether it drops useful passages or keeps unrelated ones. Filtering happens
after scoring, so it reduces the context sent to the next model, not the work
done by Jev.

### Rerank without filtering

```python
from jev_reranker import JevReranker

reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...")
results = reranker.rerank(
    "Which planet is called the Red Planet?",
    ["Venus has a thick atmosphere.", "Mars is known as the Red Planet."],
    top_k=1,
)

for item in results["results"]:
    print(item["document_index"], item["score"], item["text"])
```

`rerank()` defaults to threshold 0.0, keeping all scored candidates unless you
set `top_k` or raise the threshold. Each call closes its HTTP client automatically,
including on failure; no context manager or explicit cleanup is needed.

## Choosing between reranking and relevance filtering

| Method | Intended use | Default threshold |
| --- | --- | ---: |
| `rerank()` | Order retrieved documents by relevance | 0.0 |
| `relevance_rerank()` | Select evidence that can help answer the query | 0.2 |

The difference is the instruction and the default threshold. Both methods use
the same Jev model and scoring pipeline; `relevance_rerank()` is not simply
`rerank()` with a higher cutoff.

`rerank()` asks Jev to score how well each document helps answer the query, and
keeps all scores by default (`threshold=0.0`). `relevance_rerank()` uses a prompt
that explicitly asks for very low scores when a document only overlaps with the
topic or provides no useful evidence. This is intended to push those scores much
lower than ordinary reranking, making a threshold (`0.2` by default) more useful
for removing documents before passing context to an LLM.

That emphasis has a tradeoff. Pushing weak candidates toward zero can make their
relative order less informative, so the relevance prompt may be less suitable
than `rerank()` when the goal is to sort the entire candidate set. Use
`relevance_rerank()` when deciding which documents to keep matters most; use
`rerank()` when ordering the candidates is the main task. Both return results
sorted by descending score.

The relevance prompt still credits partial answers and concrete linking facts;
a passage need not answer the whole question to be retained. Read the
[built-in prompts](src/jev_reranker/instructions.py) and test the choice of prompt
and threshold on your own data. Lower scores and better filtering are the intent
of the instructions, not a guarantee for every document or query.

## API reference

Expand the sections below for signatures, options, and examples. Code snippets
that only show configuration assume `from jev_reranker import JevReranker`.

<details>
<summary>Requests, responses, and selection</summary>

```text
rerank(query, documents, *, instruction=None, threshold=0.0,
       top_k=None, return_documents=True, detail=False)
relevance_rerank(query, documents, *, instruction=None, threshold=0.2,
                 top_k=None, return_documents=True, detail=False)
```

`query` is a string and `documents` is a sequence of strings. The library accepts
text documents, not image payloads. All ranking methods return a dictionary
with a `results` list. The following is an illustrative shape, not a measured score:

```python
{"results": [{"document_index": 1, "score": 0.9, "text": "Original document text"}]}
```

With `detail=True`, every method adds a top-level `detail` with the complete
execution record, and each returned result also contains document-level details.
If every candidate is filtered out, the response still contains `"results": []`
and the requested top-level details, including excluded documents.

| Argument | Behavior |
| --- | --- |
| `instruction` | Override the prompt with an instructions/criteria dictionary |
| `threshold` | Keep `score >= threshold`; finite number in [0, 1] |
| `top_k` | None for all eligible results, or a nonnegative output limit |
| `return_documents` | Include original text by default; False omits it |
| `detail` | Include diagnostics; False by default |

Sort by descending score, preserving input order for ties. Duplicate texts remain
separate candidates. Text in results is untruncated. Apply the threshold before
`top_k`; top-k does not reduce scoring work. Empty input and `top_k=0` make no
requests. Unknown kwargs and invalid inputs raise errors rather than being ignored.

`relevance_rerank()` supports listwise and pointwise, not pairwise.

For full relevance diagnostics, request details:

```python
from jev_reranker import JevReranker

reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...")
response = reranker.relevance_rerank(
    "Which planet is called the Red Planet?",
    ["Mars is called the Red Planet.", "Bread is made from flour."],
    threshold=0.2,
    detail=True,
)
```

`response["detail"]["documents"]` includes excluded scores and `passes_threshold`.
`response["detail"]["selection"]` records counts before and after selection.

</details>

<details>
<summary>Authentication and optional dependencies</summary>

Install from PyPI:

```sh
uv add jev-reranker
# Or: pip install jev-reranker
```

Set `TYPESAFE_API_KEY` in `.env` or the environment. An explicit `api_key=` takes precedence over the environment, which takes precedence over `.env`. Keep `.env` out of version control. Set `dotenv_path=None` to disable file loading.

By default, lengths use Python's `len(text)`; no tokenizer dependencies or downloads are required. Select extras as needed:

```sh
uv add 'jev-reranker[all]'
# Tokenizer only: uv add 'jev-reranker[tokenizer]'
# Sentence Transformers only: uv add 'jev-reranker[sentence-transformers]'
```

The `all` extra installs tokenizer dependencies, Sentence Transformers (including PyTorch), and evaluation-only pyarrow. The base installation does not include PyTorch. Model weights are downloaded when used, not when installing the extra.

Use `api_key_env=` to read a different environment variable name.

</details>

<details>
<summary>Scoring instructions and customization</summary>

`relevance_rerank()` selects its preset by mode: the default `listwise` uses
`RELEVANCE_INSTRUCTION`; explicit `pointwise` uses `POINTWISE_RELEVANCE_INSTRUCTION`.
The latter judges each document without assuming access to other candidates.
Both default to threshold 0.2, and an explicit per-call `instruction=` takes precedence.
For the unmodified listwise relevance preset, each request stores the full
evaluation rubric once in its state. Each document question uses a shorter
reference and explicit true/false criteria. This reduces repeated prompt text;
the request budget includes both the rubric and the document questions.
This happens automatically; existing calls need no new arguments or flags.
Passing an unchanged copy of `RELEVANCE_INSTRUCTION` uses the same request format.
The preset uses an absolute evidence scale and retains partial or linking facts
while preferring evidence that addresses the requested information.
Different prompts and request groups can change scores and filtering decisions;
check the threshold on your data. Modified or custom instructions are formatted
in full for each document, and pointwise scoring keeps its single-document prompt.

`rerank()` uses `RERANK_INSTRUCTION` for listwise and pointwise scoring.
It asks whether a document helps answer the query and prefers specific facts,
without prescribing numeric score anchors.

The relevance presets add explicit anchors: topic overlap alone should receive
0.1, and unrelated content or the wrong referent should receive 0.0. Useful
partial answers and linking facts receive higher anchors. These instructions
are designed to separate useful evidence from documents to discard, rather than
preserve fine distinctions among weak candidates. As a result, they can work
better for threshold filtering while being less suitable for ordering every
candidate. Actual scores still depend on the model and context; the library
does not force scores to these anchors or rescale them after the API response.

The instruction controls scoring; the threshold controls selection after scoring.
Changing `threshold` alone does not change the prompt or the scores. For example,
`rerank(..., threshold=0.2)` keeps the ordinary ranking prompt, while
`relevance_rerank(..., threshold=0.0)` keeps the relevance prompt and returns all
scored candidates. For the same supported mode, explicit instruction, and
threshold, both methods use the same scoring and selection logic.

```python
from jev_reranker import JevReranker

reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...", mode="pointwise")
results = reranker.relevance_rerank(
    "Which planet is called the Red Planet?",
    ["Mars is called the Red Planet.", "Bread is made from flour."],
)
```

The pointwise prompt judges each document using only that document and the query.
Use `detail=True` to inspect the prompt and scores, and check the threshold when
switching between listwise and pointwise.

`RERANK_INSTRUCTION`, `PAIRWISE_INSTRUCTION`, `RELEVANCE_INSTRUCTION`, and `POINTWISE_RELEVANCE_INSTRUCTION` in [`instructions.py`](src/jev_reranker/instructions.py) are ordinary dictionaries. You can provide the same structure:

```python
custom = {
    "instructions": "Does {document} contain concrete evidence for `query`?",
    "criteria": {
        "true": "Contains concrete supporting facts",
        "false": "Contains no supporting facts",
    },
}
reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...")
results = reranker.rerank("query", ["document"], instruction=custom, threshold=0.3)
# JevReranker(api_key="YOUR-TYPESAFE-API-KEY...", instruction=custom) sets the instance default.
```

Only the keys `instructions` and `criteria` are accepted. Criteria must contain nonempty `true` and `false` strings. Templates use `{document}`, or `{left}` and `{right}` for pairwise. Unknown keys and invalid placeholders fail before any request. Each call copies its instructions without mutating instance configuration, keeping concurrent calls separate. Legacy constructor arguments `instructions=` and `criteria=` remain supported but cannot be combined with constructor `instruction=`. `relevance_rerank()` selects the relevance preset independently of the constructor's ordinary ranking prompt; its per-call `instruction=` can override that preset.

</details>

<details>
<summary>Async calls, threads, and client lifecycle</summary>

HTTP uses `httpx.AsyncClient` and `asyncio` internally:

```python
import asyncio
from jev_reranker import JevReranker

async def main():
    reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...", mode="pointwise", max_concurrency=8)
    results = await reranker.a_rerank("The Red Planet?", ["Mars.", "Venus."])
    response = await reranker.a_rerank(
        "The Red Planet?", ["Mars.", "Venus."], detail=True,
    )
    # Concurrent queries can share the same instance.
    batches = await asyncio.gather(
        reranker.a_rerank("The Red Planet?", ["Mars.", "Venus."]),
        reranker.a_rerank("Earth's satellite?", ["The Moon.", "The Sun."]),
    )
    print(results, response["detail"]["usage"], batches)

asyncio.run(main())
```

| Sync API | Async API | Return value |
| --- | --- | --- |
| `rerank()` | `a_rerank()` | Ranked results and optional execution details |
| `relevance_rerank()` | `a_relevance_rerank()` | Results passing the relevance threshold and optional execution details |
| `close()` / `with` (optional) | `await aclose()` / `async with` (optional) | Disable the instance and drain active calls |

Sync and async methods share arguments, result structure, and scoring logic, including `top_k`, `return_documents`, and `detail`.

Ordinary instances can be reused across sync calls, caller threads, and async
event loops. Each synchronous call uses a temporary `asyncio.run()` loop; no
persistent loop thread is kept. Calling a sync method inside a running loop still
raises `ConfigurationError`; use the async method instead.

Each call owns a separate, lazily created HTTP client. Success, failure, and
cancellation all close it before the call finishes. A context-local call state
keeps concurrent requests separate. No explicit shutdown is needed for normal use.

HTTP concurrency is bounded across the entire instance. Pointwise uses worker tasks, pairwise processes bounded groups of tasks, and listwise chunks run sequentially. Retries use `asyncio.sleep`. Synchronous tokenizer loading and counting use `asyncio.to_thread`.

Cancellation drains child tasks before propagating `asyncio.CancelledError`. HTTP failures also drain children and preserve public `JevError` exceptions rather than wrapping them in `ExceptionGroup`. `aclose()` and `close()` remain optional compatibility methods: they reject new work and wait for active calls, whose resources are cleaned up automatically. Cancel caller tasks first if you need to stop active work early.

</details>

<details>
<summary>Length limits and optional tokenizers</summary>

`document_max_length` defaults to 4000. The default `len(text)` counts Unicode code points, not bytes or display width. Documents exceeding the limit are sent as prefixes; returned text remains unchanged.

```python
reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...", document_max_length=8000)
results = reranker.rerank("query", ["document"])
# document_max_length=None disables document truncation.
```

For production use, consider token counting. English text in particular often has
many more characters than tokens, so character limits can truncate documents or
split requests earlier than needed. Install the optional tokenizer dependencies:

```sh
uv add 'jev-reranker[tokenizer]'
```

```python
reranker = JevReranker(
    api_key="YOUR-TYPESAFE-API-KEY...",
    tokenizer="google/embeddinggemma-300m",
    document_max_length=4000,  # Measured in Gemma tokens.
    split_state_budget=16000,
    split_request_budget=30000,
)
response = reranker.relevance_rerank("query", ["document"], threshold=0.2)
results = response["results"]
```

This example uses explicit split budgets of 16000 and 30000 for more conservative
request grouping with Gemma token counting. The defaults remain 26000 and 48000.
Gemma counts are local estimates; these settings do not guarantee that requests
fit the provider's context limit. Grouping can also affect listwise scores.

Only the tokenizer is fetched on first scoring. Gemma license acceptance and Hugging Face authentication such as `HF_TOKEN` may be required. No model weights, PyTorch, or GPU are needed. The default Gemma revision is pinned; override it with `split_tokenizer_revision="main"`, for example. Other Hub repositories, local `tokenizer.json` files, and objects implementing `encode(text)` / `decode(ids)` are supported. Supplying your own object does not require this library's tokenizer extra.

You can also provide a synchronous `length_fn: Callable[[str], int]`, such as a UTF-8 byte counter:

```python
def utf8_length(text: str) -> int:
    return len(text.encode("utf-8"))

reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...", length_fn=utf8_length, document_max_length=8000)
results = reranker.rerank("query", ["document"])  # Prefix of at most 8000 bytes.
```

The function must return a deterministic nonnegative integer and cannot be combined with `tokenizer`. It may run in a worker thread and receives both documents and serialized JSON state/questions/requests. Custom counters use binary search over text prefixes, followed by a bound check; the longest possible prefix is not guaranteed for nonmonotonic functions.

`split_state_budget` (26000) and `split_request_budget` (48000) use the same selected units. Queries are not truncated. Character counts, custom counts, and Gemma tokens are not guaranteed to match Jev's internal tokens; API context errors are handled separately. Legacy names `document_max_tokens`, `split_state_token_budget`, and `split_request_token_budget` remain aliases and do not force token-based measurement.

</details>

<details>
<summary>Scoring modes and constructor options</summary>

| Mode | Operation | Score |
| --- | --- | --- |
| `listwise` (default) | Ask about candidates in a shared state; split when limits are exceeded | Per-document score from 0 to 1 |
| `pointwise` | Ask independently for each query/document pair | Per-document score from 0 to 1 |
| `pairwise` | Compare every pair in both directions | Mean win probability, not absolute relevance |

```python
reranker = JevReranker(
    api_key="YOUR-TYPESAFE-API-KEY...",
    model="jev-latest",  # Override to select another Jev model.
    mode="listwise",
    document_max_length=4000,  # None disables prefix truncation.
    split_state_budget=26000,
    split_request_budget=48000,
)
results = reranker.rerank("query", ["document"])
```

| Option | Meaning / default |
| --- | --- |
| `model` | Explicit value → `JEV_MODEL` (environment/`.env`) → `jev-latest` |
| `api_key`, `api_key_env` | Explicit key and environment variable name (default `TYPESAFE_API_KEY`) |
| `dotenv_path` | `.env`; does not mutate the process environment |
| `endpoint` | Explicit value → `TYPESAFE_ENDPOINT` → `https://api.typesafe.ai/v1/systemone`; a complete endpoint URL |
| `max_concurrency` | 4 for listwise, 20 for pointwise/pairwise; shared instance-wide HTTP limit |
| `timeout`, `max_retries` | 180 seconds per HTTP I/O; up to 8 retries after the initial attempt |
| `split_tokenizer_name` | None; alias for a string-valued `tokenizer` |
| `split_tokenizer_revision` | Pinned commit for default Gemma; Hub default for other tokenizers unless specified |
| `tokenizer_max_length` | 65536; automatically raises a smaller supplied `model_max_length` and preserves larger values |
| `tokenizer` | None; Hub repository, local tokenizer.json, or encode/decode object |
| `length_fn` | None (uses `len` without a tokenizer); custom synchronous counter |
| `document_max_length` | 4000; None disables document truncation |
| `split_state_budget`, `split_request_budget` | 26000 / 48000 in the selected measurement units |
| `instruction` | Dictionary containing instructions/criteria; also accepted per call |
| `instructions`, `criteria` | Legacy constructor template and criteria |
| `client` | Advanced: borrowed `httpx.AsyncClient`; async methods only, on one loop; caller closes it |
| `transport` | Advanced: borrowed `httpx.AsyncBaseTransport`, such as `httpx.MockTransport`; caller closes it; cannot be combined with `client` |

Explicitly supplied clients and transports are caller-owned and are never closed
by the reranker. A borrowed client intentionally overrides per-call connection
ownership, remains bound to its first event loop, and requires async methods.
Custom transports must support the caller's concurrency and loop usage. These
advanced options are unnecessary for ordinary API calls.

Gemma length settings estimate Jev input size; they do not extend the EmbeddingGemma model's context. Queries and questions also consume the budgets. Partitioned listwise scoring uses different shared contexts and is not guaranteed to produce the same scores as an unsplit request. Pairwise uses n(n−1)/2 requests and is most suitable for small candidate sets.

Named tokenizers disable truncation and padding and count the full input even beyond 65536 tokens. Supplied objects with writable `model_max_length` are adjusted to at least `tokenizer_max_length`, mutating that object. Objects without this attribute are left unchanged. This does not change `document_max_length`.

Custom encoders should not add special tokens or truncate input. Wrap Transformers tokenizers to pass `add_special_tokens=False, truncation=False`. Custom question templates must contain `{document}` for listwise/pointwise or `{left}` and `{right}` for pairwise. Other fields and unknown kwargs are rejected.

```python
reranker = JevReranker(
    api_key="YOUR-TYPESAFE-API-KEY...",
    instructions="Does {document} directly answer `query`?",
    criteria={"true": "Provides the requested facts", "false": "Does not provide the requested facts"},
)
```

</details>

<details>
<summary>Execution details, retries, and errors</summary>

```python
import json
from pathlib import Path
from jev_reranker import JevError, JevReranker

reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...")
try:
    response = reranker.rerank("The Red Planet?", ["Mars.", "Venus."], detail=True)
except JevError as exc:
    # Inspect APIError.status_code or partial execution details in exc.detail.
    raise

Path("rerank-log.json").write_text(
    json.dumps(response, ensure_ascii=False, indent=2), encoding="utf-8",
)
```

`response["detail"]` records effective settings, Python/dependency versions, requested/resolved models, tokenizer revision, usage, retries, splits, submitted state/questions, and responses. Per-result `detail` contains `original_length`, `sent_length`, `length_unit`, related request IDs, and pairwise comparisons. `rerank(..., detail=True)` and `relevance_rerank(..., detail=True)` return the same full execution record, including excluded documents.

For the built-in listwise relevance preset, `detail["configuration"]["instructions"]`
contains the full rule template. In each entry of `detail["requests"]`, inspect
`payload["state"]["rubric"]` together with `payload["questions"]` to see the complete
submitted prompt; the per-document question alone contains only its reference.

Detail `schema_version` is 2. `usage.input_tokens` and `output_tokens` are API-reported token counts, distinct from local length estimates. Logs include document text but exclude API keys, authentication headers, and arbitrary environment variables.

HTTP 429/500/502/503/504/529 and transport errors receive exponential backoff with jitter, respecting `Retry-After` up to 60 seconds. Authentication failures and malformed successful responses are not retried. Listwise context errors repartition only the failed candidates. A document or pair that cannot fit raises `ContextLimitError`. Failures are never converted to zero scores. Other public exceptions are `ConfigurationError`, `APIError`, and `ResponseValidationError`, all derived from `JevError`.

Do not mutate constructor settings during use; create another instance for different settings.

</details>

## Evaluation

Evaluate relevance filtering or reranking on a Nano-set benchmark. Use the same
candidate selection and metrics to compare Jev with a Sentence
Transformers CrossEncoder, or inspect how a relevance threshold changes the
retained documents and positives.

```sh
uv run --locked --group examples --extra tokenizer python examples/eval.py \
  --task relevance --target en
```

Run this command from a source checkout; `examples/eval.py` is a repository script.
This example scores the full hybrid candidate pool. Add `--top-k 10` to evaluate
a controlled ten-candidate pool containing every labeled positive. See the [evaluation guide](docs/eval.md) for dataset selection, dependencies,
metrics, and output files. Evaluation calls the real Jev API and incurs usage charges.

## Contributing

Bug reports and contributions are welcome. For a ranking issue, include a minimal
reproduction, the model and mode, and the behavior you expected. Remove credentials
and private document text before sharing execution details.

Contributor guides:

- [Design and behavior](docs/spec.md)
- [Live test setup](docs/live-validation.md)
- [Repository guidelines](AGENTS.md)
- [Release guide](docs/release.md) and [changelog](CHANGELOG.md)

```sh
git clone https://github.com/hotchpotch/jev-reranker.git
cd jev-reranker
uv sync --locked --dev
uv run --locked tox
```

Normal tests do not call Jev. Live tests require explicit opt-in and credentials;
see the live test guide before running them.

## References

### HAKARI-Bench

[HAKARI-Bench](https://github.com/hakari-bench/hakari-bench) is a related project
by the same author. Its TypeSafe reranker implementation informed this library's
Jev scoring and listwise partitioning design. Its Nano-set benchmark tooling also
informed the evaluation example's hybrid candidate handling and nDCG calculation.

HAKARI-Bench also publishes the
[Nano-set benchmarks](https://huggingface.co/hakari-bench/datasets?search=nano)
used by the evaluation script. See the [evaluation guide](docs/eval.md) for
choosing a dataset and benchmark, the supported data format, and how to run
comparisons.

## License

MIT. See [LICENSE](LICENSE).

## Author

Yuichi Tateno ([@hotchpotch](https://github.com/hotchpotch)).
