Metadata-Version: 2.5
Name: ranksignal
Version: 0.1.1
Summary: Catch ranking and retrieval quality failures in production, without ground-truth labels.
Project-URL: Homepage, https://github.com/mathewOracle/ranksignal
Project-URL: Repository, https://github.com/mathewOracle/ranksignal
Project-URL: Issues, https://github.com/mathewOracle/ranksignal/issues
Author-email: Mathew Kadambatt <matmart5@gmail.com>
License: MIT
License-File: LICENSE
Keywords: evaluation,mlops,monitoring,observability,prometheus,quality,rag,ranking,recommender,retrieval,search,sre
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: hypothesis>=6.0; extra == 'dev'
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: prometheus-client>=0.15; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: prometheus
Requires-Dist: prometheus-client>=0.15; extra == 'prometheus'
Description-Content-Type: text/markdown

# ranksignal

**Catch ranking and retrieval quality failures in production — without ground-truth labels.**

[![CI](https://github.com/mathewOracle/ranksignal/actions/workflows/ci.yml/badge.svg)](https://github.com/mathewOracle/ranksignal/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/ranksignal)](https://pypi.org/project/ranksignal/)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/ranksignal/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
[![Typed](https://img.shields.io/badge/typing-strict-blue)](https://peps.python.org/pep-0561/)

Your search, RAG, or recommender pipeline returns `200 OK`. Latency is flat.
Error rate is zero. Every dashboard is green.

And the results have been garbage for six hours.

```python
from ranksignal import Monitor, Ranking

monitor = Monitor(labels={"model": "v3"})

health = monitor.observe(Ranking(results, query=query_id, requested=20))

if not health.ok:
    for report in health.degraded:
        log.warning("%s: %s", report.signal, report.reason)
```

```
starvation: returned 3 of 20 requested (15%), at or below the critical fill rate
score_collapse: scores are nearly identical (variation 0.0004); the ordering is effectively arbitrary
```

## Why this exists

Ranking quality is measured offline, with labels, in a batch job. That is a
good way to compare two models, and no way at all to find out that the one in
production broke at 2am.

| The usual approach | What goes wrong |
|---|---|
| nDCG / MRR on a labelled set | Needs ground truth. In production you have none. |
| Click-through rate | Arrives hours-to-days late, and is confounded by traffic mix |
| HTTP status and latency alerts | A perfectly fast, perfectly successful bad ranking is invisible |
| Offline eval on last month's labels | Cannot see today's index, today's model, today's bug |
| Manual spot checks | You look after someone complains |

`ranx`, `pytrec_eval` and `ir-measures` are excellent at what they do —
computing nDCG, MAP and friends against `qrels`. They are offline, batch, and
label-dependent by design. `ranksignal` is the other half: **online, per-request,
and label-free.** Use both.

## Install

```bash
pip install ranksignal                  # zero dependencies
pip install "ranksignal[prometheus]"    # metrics export
```

Python 3.9+. The core has **no runtime dependencies**.

## The idea

You cannot know whether a result is *relevant* without a label. You can know a
great deal about whether a ranking is **shaped like the broken ones**:

| Signal | Fires when | Catches |
|---|---|---|
| `Starvation` | Far fewer results than requested | Over-aggressive filters, dead shards, over-narrow query rewrites |
| `ScoreCollapse` | Every score is nearly identical | Mis-loaded model, zeroed embeddings, saturating similarity |
| `ScoreFloor` | Even the best result scores badly | Nothing in the corpus actually matched |
| `Duplication` | The same id appears repeatedly | Fan-out retrieval that forgot to de-duplicate |
| `Churn` | Little overlap with a baseline | Bad deploy, broken reindex, silent model swap |

None of these prove your results are irrelevant. They prove the ranking looks
like the ones that *were* — which is exactly what a paging threshold is for.

### Signals are scale-free where they can be

`ScoreCollapse` measures the **coefficient of variation**, not raw spread, so
cosine similarities in `0..1` and BM25 scores in the hundreds are judged on the
same axis. `Starvation` measures **fill rate**, not a raw count, so one
threshold works for a page of 10 and a page of 100.

`ScoreFloor` is the deliberate exception: "a good score" is entirely
ranker-specific, so it ships **with no default thresholds**. A plausible-looking
wrong number is worse than making you choose.

```python
from ranksignal import Monitor, ScoreFloor, Starvation

monitor = Monitor(
    signals=[
        Starvation(warn_below=0.8, critical_below=0.4),
        ScoreFloor(warn_below=0.35, critical_below=0.20),  # tuned to *your* scores
    ]
)
```

### "I couldn't measure that" is not "that's fine"

A signal that cannot run reports `SKIPPED`, never `OK`. An unscored ranking does
not quietly count as healthy, and a skipped signal is never exported as a
number — a fabricated value on a dashboard is worse than a gap in it.

```python
health.signal("score_collapse").severity  # Severity.SKIPPED
health.ok  # unaffected by skips
```

### Detecting a bad deploy

`Churn` is the one signal that needs history. Give it what the query returned
before — yesterday, or from the model you are replacing:

```python
from ranksignal import Churn, Ranking, check

health = check(
    Ranking(new_results, requested=10),
    signals=[Churn(baseline=yesterday_ids, k=10, warn_below=0.5)],
)
```

Measured as Jaccard overlap of the top-k id **sets**, because a document sliding
from rank 2 to rank 3 is noise, and a document vanishing entirely is not.

## Prometheus

```python
monitor = Monitor(labels={"model": "v3", "index": "docs-v2"})
```

| Series | Type | Use |
|---|---|---|
| `ranksignal_evaluations_total{signal,severity}` | Counter | Verdict volume |
| `ranksignal_value{signal}` | Gauge | Current quality level |
| `ranksignal_degraded_total{signal,severity}` | Counter | **Alert on the rate of this** |

```promql
# Page when retrieval starves for five minutes.
rate(ranksignal_degraded_total{signal="starvation",severity="critical"}[5m]) > 0

# Warn when fill rate sags, before anyone files a ticket.
ranksignal_value{signal="starvation"} < 0.5
```

Values are gauges because these are quality *levels*, and the natural alert is
"below 0.5 for five minutes". A histogram would answer a question nobody asks
about a bounded ratio.

**`prometheus_client` is optional.** Without it, `Monitor` still runs every
signal and still returns a `Health` — it just does not export. A monitoring
library must never be the reason a request fails, and that includes failing on
an import.

## Guarantees

Verified by property-based tests across hundreds of generated rankings
([`tests/test_properties.py`](tests/test_properties.py)):

- Evaluating a ranking **never raises** for any well-formed input.
- Evaluation **never mutates** the ranking it is given.
- Identical inputs produce **identical** verdicts.
- Every signal returns **exactly one** report — nothing vanishes unrecorded.
- A `SKIPPED` signal **never** carries a value, and **never** exports one.
- Duplicate fraction and churn overlap always land in `0..1`. Fill rate is
  deliberately *not* capped above — returning 20 when 10 were asked for is a
  fill rate of `2.0`, and clamping it would hide a real over-delivery bug.
- `ScoreCollapse` is **scale-invariant**: rescaling every score cannot change
  the verdict, which is what lets one threshold survive a swap from cosine
  similarity to BM25.

Because checking is a pure function returning a plain value, you can assert on it:

```python
def test_empty_retrieval_is_critical():
    health = check(Ranking([], requested=10))
    assert health.signal("starvation").severity is Severity.CRITICAL
```

## Development

```bash
git clone https://github.com/mathewOracle/ranksignal && cd ranksignal
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest && ruff check . && ruff format --check . && mypy
```

Contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). New signals are
especially welcome and shouldn't require touching anything but your own class.

## License

MIT — see [LICENSE](LICENSE).
