Metadata-Version: 2.4
Name: notorious
Version: 0.0.1
Summary: Build a retrieval eval for your own corpus: pool candidates, LLM-judge them under a rubric you own, and measure judge-vs-human alignment before trusting any metric.
Project-URL: Homepage, https://github.com/georgian-io/notorious
Project-URL: Repository, https://github.com/georgian-io/notorious
Project-URL: Issues, https://github.com/georgian-io/notorious/issues
Author-email: Georgian <paul@georgian.io>
License-Expression: MIT
License-File: LICENSE
Keywords: evaluation,information-retrieval,llm,ndcg,rag,ranking,relevance-judgments,retrieval,search
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.10
Requires-Dist: ir-measures>=0.3.3
Requires-Dist: jinja2>=3.1
Requires-Dist: litellm>=1.40
Requires-Dist: numpy>=1.24
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: jsonschema>=4.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: pytrec-eval-terrier>=0.5.10; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Description-Content-Type: text/markdown

# Notorious NDCG

**Did that change to your search actually make it better?** Answer it with an LLM-graded answer key
you've checked you can trust — then compare your retrieval configurations using standard search metrics.

## Why you'd use this

You changed something in your search or RAG stack — swapped the embedding model, resized your chunks,
added a reranker, retuned your hybrid-search weights. The question is always the same:

> Did retrieval actually get better, or did the numbers just move around?

Most teams answer by eyeballing a handful of queries. That doesn't scale, and it quietly misses
regressions: a change that helps ten queries and hurts twenty can still "look fine."

Answering it properly needs an **answer key** — for a set of test queries, which documents in your
corpus are actually relevant. With that key you can compute real search-quality scores and compare
configurations head to head. The catch: a good answer key is expensive to build by hand, and the obvious
shortcut — *"just ask an LLM which documents are relevant"* — trades one problem for another. Does the
LLM's notion of *relevant* match yours? If you can't answer that, every metric built on top of it is
standing on sand.

`notorious` is built around closing exactly that gap:

1. It uses LLMs to grade relevance at scale, so you don't hand-label thousands of query–document pairs.
2. Before trusting those grades, it makes the agreement measurable. You review a small sample, correct
   the grades you disagree with, and the tool reports how often the judge matched you — with a confidence
   interval, so you know when you've checked enough.
3. You tune a **rubric** (your written definition of "relevant") until the judge reproduces your
   corrections on its own. Then you freeze it and score the full test set.

What you get out: a portable **TREC-style qrels** file (the answer key) and, per configuration,
**NDCG@k, Recall@k, and MRR** with error bars across multiple judge models. Everything is plain files on
disk you can open, diff, and re-run.

Those metric names are opaque until someone tells you what they mean:

- **NDCG@k** — of the top *k* results, are the most-relevant ones near the top? (overall ranking quality)
- **Recall@k** — of all the relevant documents, how many made it into the top *k*?
- **MRR** — on average, how high up the list is the *first* relevant result?

## Install

```bash
pip install notorious          # or: uv pip install notorious
notorious --help
```

Prefer an isolated tool install:

```bash
uvx notorious --help           # run without installing
pipx install notorious         # or install it as a standalone CLI
```

### From source (for development)

```bash
git clone https://github.com/georgian-io/notorious
cd notorious
uv sync                        # create the environment
uv run notorious --help
```

## Quickstart

The scaffold ships **mock, deterministic judge models**, so you can run the entire loop offline and
free — before spending a cent or wiring up your own search:

```bash
notorious init my-eval        # scaffold a runnable project: toy corpus, rubric, configs, mock judges
notorious run my-eval --yes   # pool → judge → report, all at once
notorious review my-eval      # open the browser page to grade the judge
```

What to expect:

- The mock judges need no API key and cost nothing; every artifact lands in `my-eval/out/`.
- `review` is keyboard-first and **blind** — the judge's grade reveals only *after* you commit yours,
  which is what keeps the agreement number honest.
- Lost the thread? **`notorious status my-eval`** prints the current state and the exact next command
  to run. It never writes anything, so run it any time.

New to the idea? The **[walkthrough](./docs/examples/walkthrough/)** narrates this same run end to end
with screenshots. Ready for your own stack? Skip to
[Bringing your own corpus and configs](#bringing-your-own-corpus-and-configs).

## How it works

### The four stages

There are four stages. Each one reads and writes a file on disk you can open, hand-edit, diff, or re-run
on its own:

```
queries.jsonl ──► pool ──► judge ──► report
```

**`pool`** — for every test query, runs each of your configs and collects the union of their top-*k*
results into one candidate set (`out/pool.jsonl`). This is *pooling*: rather than grade your entire
corpus against every query (impossible at any real scale), you grade only the documents your configs
actually surface. Deduped by `doc_id`, with each config's rank recorded.

**`judge`** — asks each judge model to grade every `(query, document)` pair in the pool on a **0–3
relevance scale**, with a written rationale, following your `rubric.md`. Grades are cached by content, so
a re-run only calls the model for genuinely new pairs and an interrupted run resumes with no lost work.
Calls run concurrently (`judge_concurrency`, with an optional global `judge_rate_limit_per_min`).

The prompt treats the query and document as **untrusted data** — the model is told to ignore any
instructions embedded in them, and delimiter tags inside the text are defanged — so a document can't talk
the judge into a grade.

**`report`** — turns the grades into the answer key and the metrics. It writes:

- `out/qrels.txt` — the TREC-style answer key;
- per-config **NDCG@k / Recall@k / MRR** — plus **P@k** as a diagnostic (exact, not pool-relative, since
  every top-*k* doc is judged);
- `out/report.md` — a readable summary, led by a **decision box** (run status, judge trust,
  comparison verdict, the primary blocker, and the one next action);
- `out/report.json` — the same metrics, significance, alignment, and diagnostics as stable JSON, for CI
  gates and dashboards (no prose scraping);
- `out/review.html` — the interactive review page.

With multiple judges, every metric carries the spread across them as error bars. `report` also runs a
**paired per-query significance** test between every pair of configs under test — the mean NDCG@k
difference with a bootstrap confidence interval — so you can tell a real win from query-sampling noise,
which is the whole "did it actually get better?" question. A comparison whose interval is too wide to
decide is flagged **inconclusive** (rather than a tie), which is the signal to judge more queries — a
small inner-loop sample earns judge trust, but the decision usually wants the larger frozen run.

**`review`** — serves the review page from a local, loopback-only server (`http://127.0.0.1:8765`),
rendered live from the current files. This is where you grade the judge card by card (see
[Aligning the judge](docs/aligning-the-judge.md)). Grading is keyboard-first —
**`0`–`3`** to grade, **`←`/`→`** to move, **`u`** to jump to the next ungraded card — and a session
panel tracks your graded count, grade distribution, and relevant-vs-not balance as you go.

### Every run records its own provenance

Each of `pool`, `judge`, and `report` also writes `out/run-manifest.json` — a provenance sidecar
recording the tool version, hashes of every input (`run.yaml`, rubric, queries, configs, pool), the judge
set, the judgment split, and when each stage last ran. It complements the human-readable report: it's the
one file that answers "what exactly produced these numbers?" without reconstructing it from the others.

### Convenience and inspection commands

- **`notorious run`** chains `pool → judge → report` in one go, with the same cost confirmation as `judge`
  (`--yes` skips it). It stops on the first stage that errors and prints the next human action. It's an
  on-ramp, not a replacement — every stage stays independently runnable, which is what the inner/outer
  loop relies on.
- **`notorious status`** is a read-only inspector for when you've lost track of where you are. It prints
  which artifacts exist, the current rubric hash and query-provenance mix, how the pool lines up with your
  queries, how many judge calls a `judge` run would make now (and how many are cached), the standing
  corrections and their alignment CI — and, from all of that, the single command to run next. It reuses
  the pipeline's own bookkeeping, so its counts are exactly what the next stage will act on. Run it any
  time; it never writes anything.
- **`notorious doctor`** preflights the configured judge models and query generator **offline** before a
  real (paid) run — no calls made. For each model it reports whether its API key is visible to LiteLLM and
  whether LiteLLM knows its price, and exits non-zero if any model is missing a key. Mock models are
  always ready.
- **`notorious validate`** is the retrieval-side counterpart to `doctor`: it imports your `configs.py` and
  calls each config's `search` on a few queries (no LLM calls), checking every result is a well-formed
  `(doc_id, text)` list with unique, non-empty ids — and reports each config's latency and a few sample ids.
  It exits non-zero on a malformed adapter, so a wiring bug surfaces here instead of mid-`pool`.

## The trust loop — earn it, then spend it

The heart of the tool is a two-loop workflow:

- **Inner loop (small, cheap, interactive).** On a sample of ~20–50 queries, grade cards *blind* against
  your rubric, watch how often the judge agrees — with a confidence interval — and edit `rubric.md` until
  agreement is high and the interval is tight. This *earns* trust in the judge.
- **Outer loop (large, frozen, repeatable).** Freeze the rubric, point at the full query set, and score
  every config with the trusted ensemble. This *spends* that trust on a decision — and you re-run it on any
  change (new embedding model, reranker, chunking tweak) to watch the metrics move.

→ **[Aligning the judge](docs/aligning-the-judge.md)** walks through both loops in detail: grading blind,
the alignment score's two levers (interval *width* = "reviewed enough?", *level* = "does the judge agree
with me?"), why you need two-sided corrections, and the ensemble-transfer check.

## Reading the report

The report leads with the decision: for every pair of configs under test, is one a **real winner**, a
**genuine tie**, or **inconclusive** (underpowered — go judge more queries)? Backing that up are three
numbers that all look like error bars but answer three *different* questions, computed with different math
on purpose — treating them alike is the most common way to misread a report.

→ **[Reading the numbers](docs/reading-the-numbers.md)** explains all three (cross-judge spread, the
Wilson alignment interval, the bootstrap significance interval) and why a *tie* is a decision while
*inconclusive* is a to-do.

## Bringing your own corpus and configs

The tool **never ingests, embeds, chunks, or indexes anything** — you keep your retrieval stack and expose
each configuration you want to compare as a small adapter. A "config" is any object with a `search` method
that takes a query and returns ranked `(doc_id, text)` pairs, best result first:

```python
# my-eval/configs.py
class EmbeddingSearch:
    def __init__(self, model): ...
    def search(self, query: str) -> list[tuple[str, str]]:
        # call your vector DB / search service; return results best-first
        return [("doc-42", "text of doc 42"), ("doc-7", "text of doc 7")]

# Each entry is one configuration you're comparing — a "row" in the report.
CONFIGS = {
    "embed-large": EmbeddingSearch("text-embedding-3-large"),
    "embed-small": EmbeddingSearch("text-embedding-3-small"),
}
```

A config can be anything behind that method — a raw embedding search, hybrid fusion, a reranker stacked on
a retriever, query rewriting. List which configs to compare in `run.yaml`, and point `judge_models` at
real LLMs (e.g. `openai/gpt-4.1`, `anthropic/claude-sonnet-5`, `gemini/gemini-2.5-pro` — three are
recommended, as above).

### No test queries yet, but you have a corpus? Bootstrap some.

```bash
uv run notorious generate-queries my-eval --chunks corpus.jsonl --per-chunk 2
```

This drafts questions each chunk could answer (via `query_gen_model`), tagged `generated`. By default they
land in a **separate `queries.generated.jsonl`** — a draft to review, not answer-key evidence — leaving
your eval set untouched; pass `--append` to fold them straight into `queries.jsonl`, or `--out FILE` to
name a target. It shows a cost estimate (with the same confirmation as `judge`) before calling, and drops
near-duplicate drafts.

Once promoted, generated queries flow through the **same** pool → judge path as real ones — the source
chunk is never treated as the answer key — and every report prints the provenance mix, so you can see how
much of your eval rests on generated vs. real queries.

### Reviewing without the local server

Corrections can also be saved without `notorious review`: open the static `out/review.html` as a plain
file, and Save uses the File System Access API where available, or downloads a corrections file that
`notorious merge-corrections my-eval corrections.jsonl` folds back into the standing set.

## Which files carry state

Both loops are the same pipeline over different query sets. In short: `queries.jsonl` is the working set
(whatever it holds is what gets pooled, judged, and reported); `out/pool.jsonl` is derived and disposable;
and the durable, content-addressed stores are `out/corrections.jsonl` (your ground truth, which survives
rubric edits) and `out/judgments.jsonl` (the judge cache). Corrections are single-reviewer, last-write-wins.

→ **[Queries and artifacts](docs/queries-and-artifacts.md)** has the full rules: sampling an inner-loop
subset, cache maintenance (`cache stats|compact|prune`), the single-reviewer model, and why a rubric edit
re-judges the whole pool.

## What's included

The full pipeline is complete and hardened for honest comparisons:

- **The core loop** — `pool` → `judge` → `report`, plus the interactive review/alignment page and
  multi-judge ensembles that report every metric as mean ± spread across judges.
- **Comparison-validity guards** — the pool stage refuses when two configs return the same `doc_id` with
  different text (they must retrieve from the same corpus snapshot), and chunking experiments can declare
  parent documents via structured ids (`id_separator`) instead of a hand-maintained `parent_map`.
- **Diagnostics in every report** — a pool-diversity measure (are your configs blind in the same way?), a
  grade-concentration tripwire that flags when the 0–3 scale has stopped separating your configs, and
  paired per-query significance (with a bootstrap CI) on the NDCG gap between every two configs under test,
  flagging a win, a genuine tie, or an **inconclusive** (underpowered) comparison that needs more queries.
- **Pool wideners** — `pool_only` configs (e.g. a lexical/BM25 retriever) broaden the candidate set so a
  pool of similar configs can't look falsely complete. `init` writes an in-memory BM25 example into your
  configs module; point it at your own server-side lexical search at scale. When a widener is present, the
  report adds an **experimental pool-completeness estimate** — capture-recapture (Lincoln–Petersen) over
  how much the dense configs and the lexical widener overlap on relevant docs, giving a rough "what
  fraction of the relevant population is in the pool?" with a CI. It's a heuristic with stated assumptions,
  offered *alongside* the pool-relative recall caveat, never as a replacement for it.
- **Practical judging** — concurrent calls with an optional global rate limit, content-addressed caching
  that resumes interrupted runs, and an optional query generator (`generate-queries`) that drafts queries
  from corpus chunks.

Judging is **pointwise** — each pair graded independently on 0–3. That is the only mode implemented; a
comparison-based (pairwise) mode is a possible future direction, not a shipped feature, and would take
more than a new labeler subclass (the pipeline plans, caches, and costs one pair at a time). The
grade-concentration diagnostic is what would tell you, on evidence, that this corpus needs one.

## Documentation

Deeper guides live in **[`docs/`](docs/)**:

- [Aligning the judge](docs/aligning-the-judge.md) — the inner/outer trust loop in detail.
- [Reading the numbers](docs/reading-the-numbers.md) — the three intervals and the decision they support.
- [Queries and artifacts](docs/queries-and-artifacts.md) — which files carry state; cache maintenance.
- [The `report.json` contract](docs/report-json.md) — machine-readable report, JSON Schema, CI-gating recipes.
- [Examples](docs/examples/) — runnable projects, from an offline walkthrough to real-corpus benchmarks.
- [Development](docs/development.md) — set up the environment, run the checks, build the package.
- [Contributing](CONTRIBUTING.md) — conventions for pull requests.
