Metadata-Version: 2.5
Name: verity-retrieval
Version: 0.1.1
Summary: Measure retrieval, don't assume it. Deterministic metrics, and filtered vector search that actually returns k results.
Project-URL: Homepage, https://github.com/Raghu23-dev/verity
Project-URL: Repository, https://github.com/Raghu23-dev/verity
Project-URL: Issues, https://github.com/Raghu23-dev/verity/issues
Author-email: Raghuram P <raghu2308.dev@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: ann,evaluation,hnsw,ndcg,rag,recall,retrieval,vector-search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: numpy>=1.26
Provides-Extra: dev
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# verity

**Measure retrieval, don't assume it.** Deterministic metrics, and filtered vector search that
actually returns *k* results.

[![PyPI](https://img.shields.io/pypi/v/verity-retrieval)](https://pypi.org/project/verity-retrieval/)
[![CI](https://github.com/Raghu23-dev/verity/actions/workflows/ci.yml/badge.svg)](https://github.com/Raghu23-dev/verity/actions/workflows/ci.yml)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Coverage 98%](https://img.shields.io/badge/coverage-98%25-brightgreen)](#development)
[![Types: strict](https://img.shields.io/badge/mypy-strict-blue)](#development)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/Raghu23-dev/verity/badge)](https://scorecard.dev/viewer/?uri=github.com/Raghu23-dev/verity)

---

## Two problems

### 1. Filtered vector search silently returns fewer results than you asked for

Ask a vector store for *the 10 nearest neighbours where `lang = 'py'`* and most of them do this:

```python
candidates = index.search(query, k=10)             # the filter is ignored here
return [c for c in candidates if c.lang == "py"]   # and applied here
```

If only 3 of the 10 nearest neighbours are Python, you get **3 results** — not the 10 nearest
Python documents. No error. The request looks successful. pgvector has these semantics, and so
does the default HNSW path in several other stores.

It gets worse as the filter gets more selective, and it hides from the metric most people check:
`precision@10` over 3 results that happen to be correct scores **1.0** if you divide by what came
back instead of by *k*.

### 2. Retrieval quality is measured by asking a language model

Every major RAG framework evaluates retrieval *indirectly* — an LLM judges whether the retrieved
context looks relevant. That buys a judge with a **13.6% flip rate** on repeated runs, to
approximate quantities that are exact, free, and standard since the 1990s.

An LLM judge is the right tool for *is this answer good*. It is the wrong tool for *did the
retriever return the document I already know is relevant* — that question has a ground truth, and
comparing against ground truth doesn't require judgement.

---

## Measured, not asserted

`verity bench` produces this table. Fixed seed, exhaustive oracle in the same process, so the
numbers reproduce on your machine:

| selectivity | matching | mode | returned / 10 | recall | short |
|---|---|---|---|---|---|
| 1% | 21 | `post` | 0.12 | 0.012 | 100% ⚠️ |
| 1% | 21 | `pushdown` | 10.00 | 1.000 | 0% |
| 1% | 21 | `pre` | 10.00 | 1.000 | 0% |
| 3% | 79 | `post` | 0.36 | 0.036 | 100% ⚠️ |
| 3% | 79 | `pushdown` | 10.00 | 1.000 | 0% |
| 3% | 79 | `pre` | 10.00 | 1.000 | 0% |
| 10% | 228 | `post` | 1.10 | 0.110 | 100% ⚠️ |
| 10% | 228 | `pushdown` | 10.00 | 1.000 | 0% |
| 10% | 228 | `pre` | 10.00 | 1.000 | 0% |
| 30% | 611 | `post` | 3.14 | 0.314 | 100% ⚠️ |
| 30% | 611 | `pushdown` | 10.00 | 1.000 | 0% |
| 30% | 611 | `pre` | 10.00 | 1.000 | 0% |
| 100% | 2000 | `post` | 10.00 | 0.786 | 0% ⚠️ |
| 100% | 2000 | `pushdown` | 10.00 | 0.786 | 0% ⚠️ |
| 100% | 2000 | `pre` | 10.00 | 1.000 | 0% |

2,000 documents · 128 dimensions · k=10 · 50 queries · `degree=16, ef_search=64` · seed 42.
**short** is the fraction of queries that came back with fewer than 10 results.

Read the `post` row at 3% selectivity: **0.36 results out of 10, on 100% of queries.** The correct
answer has 10, and `pre` proves it does. That is a RAG pipeline answering from almost nothing while
reporting success.

`pushdown` returns 10 of 10 with perfect recall at every selectivity. At 100% — where the filter
matches everything, so there is nothing to get wrong — `post` and `pushdown` converge to the same
0.786, which is ordinary approximate-search behaviour and confirms the gap at lower selectivities
comes from the filtering rather than the index.

**Why `pushdown` works:** push the predicate *into* the graph traversal. A candidate that fails
the filter is still a useful stepping stone — its neighbours may pass — so it is traversed but not
collected. That is the idea behind ACORN-style predicate-aware search, and it is the whole
difference between "3 of 10" and "10 of 10".

---

## Use it

```bash
pip install verity-retrieval    # the import name is `verity`
```

Vectors may be plain lists — an embedding API returns JSON, so that is usually what you
have — or numpy arrays. Either way they are L2-normalised on the way into an index.

**Prove your store has the bug.** Compare its filtered results against an exhaustive oracle:

```python
from verity import BruteForceIndex, GraphIndex, FilterMode, Record, recall_loss

oracle = BruteForceIndex(records)          # exhaustive, therefore correct
graph  = GraphIndex(records)

exact = oracle.search(query, k=10, predicate=is_python)
actual = your_store.search(query, k=10, filter={"lang": "py"})

recall, shortfall = recall_loss(actual, exact)
if shortfall:
    print(f"asked for 10, got {10 - shortfall}: your store post-filters")
```

**Score a retriever against a golden set:**

```python
from verity import Query, evaluate

queries = [Query("q1", "how does auth work?", relevance={"auth.py": 3.0, "session.py": 1.0})]
ev = evaluate(my_retriever, queries, k=10)
print(ev.summary())
print(ev.worst(5))          # the queries to go and debug
```

**Fuse hybrid results:**

```python
from verity import reciprocal_rank_fusion

fused = reciprocal_rank_fusion({"bm25": bm25_ids, "vector": vector_ids}, limit=10)
fused[0].contributions      # {'bm25': 1, 'vector': 3} — which retriever put it there
```

---

## Design decisions

**`recall_at_k` returns 0.0 when nothing is relevant, not 1.0.** Some libraries return 1.0 ("we
found all zero of them"), which lets an unlabelled query inflate an average into looking perfect.
The harness counts and *excludes* those queries instead, and reports the count — a golden set
that's 30% unlabelled is a fact about your evaluation, not about your retriever.

**Shortfall is reported separately from recall.** They are different failures needing different
fixes: imperfect ranking is normal for an approximate index; returning fewer results than
requested is a contract violation. Averaging them together hides the second inside the first.

**`precision_at_k` divides by k, not by what came back.** This is what makes the bug above visible
instead of flattering.

**RRF sums contributions; it doesn't take the best rank.** A surprisingly common implementation
does `scores[doc] = max(scores[doc], 1/(k+rank))`, which discards exactly the cross-retriever
agreement RRF exists to capture. There's a test asserting the sum.

**No reranker.** The evidence is weaker than its popularity: published comparisons put BM25 alone
at 0.662 nDCG@10 in ~0.1 ms against 0.671 with a cross-encoder at ~225 ms — a gain inside the
confidence interval for roughly 2000× the latency. verity gives you the fusion and the measurement;
budget the rerank yourself, on your own corpus.

**One dependency: numpy.** No vector database, no embedding provider, no LLM client. A judge model
in the dependency tree would undercut the entire argument, and it means the test suite runs offline
with no credentials.

---

## A bug this project found in itself

The first benchmark run showed `pushdown` degrading badly at higher selectivity — 86% shortfall at
10% — and unfiltered recall of only 0.36. The graph was fine. The **search was quitting early**: I
had conflated the result set with the search frontier and terminated on a visit budget derived from
it.

Separating them, as HNSW does — a bounded result heap, with termination when the nearest unexplored
candidate is worse than the worst result held — moved unfiltered recall from **0.360 → 0.780** at
`ef=64`, and to **1.000** at `degree=32, ef=128`.

Worth recording for two reasons. It is exactly the failure this library exists to catch: without an
exhaustive oracle to measure against, "recall 0.36" is indistinguishable from "this is just how ANN
works". And the fix is in `TestTraversalQuality`, which asserts recall is monotonic in `ef_search`
and reaches >0.9 — so it can't come back.

---

## What this is not

- **Not a production vector store.** `GraphIndex` holds vectors in memory and doesn't persist. It
  is a real navigable small-world graph — one layer, exact neighbour construction — so the
  comparison isn't a strawman, but use it to *prove* your store has this bug, then fix your store.
- **Not an answer-quality evaluator.** verity measures retrieval. Whether the generated answer is
  good is a different question, and one where an LLM judge is appropriate.
- **Not a RAG framework.** No chunking, no embedding, no orchestration.

---

## Development

```bash
uv venv && uv pip install -e ".[dev]"
uv run pytest              # 90% coverage floor, enforced
uv run mypy src/verity     # strict
uv run ruff check .
uv run verity bench        # reproduce the table above
```

86 tests, 98% coverage, mypy strict, doctests in CI, green on Python 3.11–3.13.

Correctness is checked three ways: unit tests per metric and mode; **property-based tests**
(Hypothesis) asserting metrics stay in [0,1], recall is monotonic in k, and pushdown never
under-returns when k matching documents exist; and the **benchmark itself**, whose shape is
asserted — post-filtering must under-return and pushdown must not, or the suite fails.

## Licence

MIT
