Metadata-Version: 2.4
Name: retrieval-fairness
Version: 0.1.0
Summary: Exposure audit for vector search / RAG: coverage, dark matter, Gini, hub capture, CI gate
Author: derishabl
License: MIT
Keywords: rag,vector-search,retrieval,fairness,audit,exposure,coverage,dark-matter,hubness,retrievability,antihub
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.24
Requires-Dist: scikit-learn>=1.3
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Provides-Extra: faiss
Requires-Dist: faiss-cpu>=1.7; extra == "faiss"
Provides-Extra: pgvector
Requires-Dist: psycopg[binary]>=3.1; extra == "pgvector"
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.7; extra == "qdrant"
Provides-Extra: models
Requires-Dist: sentence-transformers>=2.2; extra == "models"
Provides-Extra: fastembed
Requires-Dist: fastembed>=0.2; extra == "fastembed"

# retrieval-fairness

**Exposure audit for vector search / RAG.** Shows what share of your
vector corpus is actually reachable by queries and what share is never
retrieved (**dark matter** — the antihub inventory of your index);
measures exposure concentration (Gini), hub capture, and regression
diff when you change the embedder or chunking. Think of it as a
health/coverage report for your *index*, not your pipeline.

> Status: early development. This is **packaging novelty** (the Gini /
> retrievability metrics are honestly borrowed from IR-fairness /
> T-Retrievability research), not a from-scratch invention.

## Why your index has dark matter (it's not a bug in your code)

Hubs and never-retrieved chunks are a **structural property of
high-dimensional nearest-neighbor search**, not a symptom of a bad
embedder. Radovanović, Nanopoulos & Ivanović (JMLR 2010, [“Hubs in
Space”](https://www.jmlr.org/papers/v11/radovanovic10a.html)) showed
that as dimensionality grows, the distribution of k-occurrences becomes
strongly right-skewed: a few points (hubs) appear in a disproportionate
share of neighbor lists, while others (antihubs) appear in almost none.
Modern ANN indexes (HNSW) amplify the effect — hub nodes are the
highway entry points that make the graph navigable. Left unmeasured,
this silently collapses the *effective* size of your corpus. This tool
makes it measurable — and gateable in CI.

## Installation

```bash
pip install -e .            # or pip install retrieval-fairness
```

## Quick start

```bash
retrieval-fairness demo --top-k 5          # demo on a synthetic corpus
retrieval-fairness demo-diff --top-k 5     # regression diff for an embedder migration
```

## Usage

### Run against real queries

```bash
# corpus.jsonl: {"id": "...", "text": "...", "vector": [...]}
# queries.jsonl: {"id": "...", "vector": [...]}
retrieval-fairness probe --corpus corpus.jsonl --queries queries.jsonl \
    --top-k 10 --json report.json --html dashboard.html
```

### No query logs: antihub self-query audit (synthetic queries)

For each chunk, generate the query that *should* retrieve it (its own
top TF-IDF terms) and check whether it actually surfaces. A chunk that
cannot be found even by a query aimed at it is invisible from any
reasonable query direction — dark matter from day one:

```bash
retrieval-fairness synth --corpus corpus.jsonl --top-k 10 --html dashboard.html
```

### Regression diff (embedder/chunking change)

```bash
retrieval-fairness diff --baseline before.json --candidate after.json
```

### CI gate

```bash
retrieval-fairness gate --baseline v1.json --candidate new.json --strict \
    --max-coverage-drop 0.05 --max-dark-matter-rise 0.05
# exit 1 in strict mode if coverage dropped > 5 pp -> CI blocks the deploy
```

### Cross-check dark matter against qrels ("lost gold")

If you have relevance judgments (qrels), cross-check which dark-matter
chunks are actually relevant — the corpus contains material the
retriever never surfaces. No competitor exposure tool ships this:

```bash
retrieval-fairness qrels --probe report.json --qrels qrels.json \
    --queries queries.jsonl --json qrels_report.json
# prints: lost gold count, recall@k, dark_relevant_ids
```

## Metrics

| Metric | What it shows |
|---|---|
| Coverage % | share of the corpus retrieved at least once |
| Of reachable ceiling % | coverage as a share of what the workload can physically reach (n_queries × top_k); distinguishes a bad retriever from a small workload |
| Dark matter % | share NEVER retrieved |
| Gini | exposure concentration (0 = uniform, 1 = all in one) |
| Hub capture top5/10 | share of exposure captured by top-N hubs |
| Lorenz curve | inequality visualization |
| Per-query overlap | result stability across a migration |
| Lost gold / Recall@k | dark-matter chunks that are actually relevant (qrels cross-check) |

## How it works

- `retrieval_fairness/types.py` — the `VectorStore` contract (Protocol).
  Any store (FAISS, Qdrant, Pinecone, pgvector) is bridged to it via
  an adapter.
- `stores.py` — `InMemoryVectorStore` for dev/tests/demos.
- `metrics.py` — coverage, gini, lorenz, hub_capture, FairnessReport.
- `probe.py` — run a workload → retrieval frequency → report.
- `diff.py` — regression diff between two runs.
- `gate.py` — CI gate with configurable rules.
- `synth.py` — synthetic queries generated from the corpus.
- `dashboard.py` — self-contained HTML report (Lorenz, histogram, PCA map).

Real-scale case study (BEIR NQ, ~50% dark matter, lexical→dense
regression diff): `docs/case_study_nq.md`. Store adapters:
`docs/adapters.md`. Comparison with related work: `docs/comparison.md`.

## Tests

```bash
pytest tests/ -q
```

## License

MIT
