Look inside a retrieval ranking

00Orientation: the questions after the metric

A search system, a RAG retriever, or a "find the matching document" feature produces a ranked list. Teams summarise it with one number, say recall@10 = 0.53, and decide from that. rankscope exists because that number hides four things: whether it is stable, which class of query fails, which component found each answer, and whether the system should have said "not found". Everything below is an instrument for one of those.

Vocabulary

lane

one retrieval system's ranked output for every query. A TREC run file is a lane.

qrels

graded relevance judgments. A query judged with no positive grade is a negative: the corpus holds no answer for it.

doc / group

two levels of "right": the exact passage, or any passage of the right document.

stratum

a label per query, so tables can show the class an average hides.

window

how many candidates the next stage sees. Attribution is measured inside it.

floor

a score below which the top hit is not evidence that the answer exists.

01Lanes and judgments

A TREC run line is qid Q0 docid rank score tag. rankscope orders the hits of a query by score, descending, breaking ties on the doc id, exactly as trec_eval does; the rank column is not trusted. A qrels line is qid 0 docid grade. Edit either box and the parsed view updates.

02A rank has a level

rank_of returns the 1-based position of the first relevant hit, and 0 for a miss. Zero is a sentinel: every consumer tests 0 < r <= k. With a group function, the same ranking has a second rank: the first hit whose group contains a relevant document. Group level answers "right document?"; doc level answers "right passage inside it?". They fail differently and want different fixes.

03hit@k, recall@k, MRR and nDCG

Four metrics, four different questions. hit@k: did any relevant document make the top k. recall@k: what fraction of the relevant documents did. MRR: how deep is the first one. nDCG@k: how good is the whole top k, weighting graded relevance by position. hit@k is a proportion and gets a Wilson interval; the others are means, so the report gives them a percentile bootstrap interval (lesson 12 shows the machinery), and two systems are compared with a paired bootstrap.

04The Wilson interval and the rule of three

A hit rate of 8 out of 12 is not "67 percent". It is "somewhere between 39 and 86 percent". The Wilson score interval says so without collapsing to zero width at 0 or n out of n, which the textbook normal approximation does. After zero events in n trials the upper limit is z²/(n+z²), which is where the table of "how many negatives do I need" comes from.

05Strata: what the average hides

A stratum is a label per query. rankscope reports every table per stratum and overall, because a mean can be respectable while one class of query is at zero. The stratum you name with --blind is the one the attribution block singles out: how many of a lane's misses fall in it.

06Miss attribution: whose fault is a miss

For each positive, given every lane's doc rank and the window: a relevant document absent from the corpus is not_indexed (coverage; no lane can help). No lane inside the window is not_surfaced (retrieval). Otherwise the label names the lanes that carried it. The share of positives a new lane surfaces that no other lane does is the number that says whether it earns its place.

07Reciprocal rank fusion, and what it dilutes

Presence in two lanes beats being first in one: rank 1 in a single lane scores 1/61 ≈ 0.0164, while ranks 3 and 5 in two lanes score 1/63 + 1/65 ≈ 0.0313. That is the point of RRF and also its cost: a lane's confident-wrong top hits can dilute another lane's correct top hit. Small k trusts the top of each lane; large k trusts presence. The instrument opens on exactly that case.

08Convex fusion, and fitting weights without lying

Each lane's scores are min-max normalised over its own list (absent document = 0, constant list = all ones), then summed with weights on the simplex. calibrate walks a grid of weight vectors and keeps the one with the best mean objective over the calibration queries. The instrument fits on one synthetic query set and evaluates on another, then lets you cheat.

09The null floor

Run the negatives, the queries whose answer is not in the corpus, and record the best score each one reaches. Over those n maxima, floor = mean + 3·std. A top score below the floor is relative evidence only: it orders candidates but says nothing about whether the answer exists. The floor is a property of the negatives you fed it, which is the whole lesson: negatives from another domain give a floor that looks safe and is not.

10The conformal floor, and how few is too few

A finite-sample guarantee on misses, with one catch: when the index exceeds n there is no finite threshold with that guarantee. rankscope returns +inf (a floor of −inf: accept everything, decide nothing) instead of quietly using the sample minimum, whose coverage is only n/(n+1). And a conformal floor bounds misses; it says nothing about false accepts.

11The gate: FNIR at a fixed FPIR

A wrong accept counts as a miss, the 1:N identification convention. sweep evaluates every floor the observed top scores suggest, in linear time: the scores a floor alone would decide are sorted once and each floor is a bisection, so a sweep over thousands of queries takes a fraction of a second rather than the twenty seconds a re-evaluation per floor did. operating_point picks the lowest floor whose FPIR is within a target, on the point estimate or, with --conservative, on the Wilson upper bound. Picked on calibration negatives it is a threshold; picked on the negatives you report it is tuning. The curve below uses the samples from lesson 9.

Two more conditions live in the same gate. The margin is the gap between the top score and the runner-up; margin_sweep does for that gap what the floor sweep does for the score, and calibrate writes an operating margin beside the operating floor. A lone candidate has an infinite gap and passes every margin, which is why a margin alone cannot drive FPIR to zero. Mutual nearest neighbour reads the ranking in the other direction: from the lanes, rankscope builds, for every document, the queries that retrieved it, best first; a gate with mutual = r accepts only when the query is within the top r of that list for its own top document. It is cheap, needs no scores to be comparable across queries, and is vacuous on a tiny query set.

12Paired bootstrap: is B really better than A?

Two systems scored on the same queries give one delta per query. Resample the queries with replacement, average the deltas each time, and read the 95 percent interval and P(delta > 0). rankscope calls it better only when P ≥ 0.95 and the mean delta clears a minimum effect size you choose, because with enough queries a delta of 0.001 becomes "significant" and still means nothing. Pairing removes query difficulty as a noise source; the toggle shows how much.

13Pre-registered verdicts

A rule is a dotted path into the report, >= or <=, and a threshold. It passes on the point when the value clears the threshold and on the interval when the conservative end of the Wilson interval does. A rule may instead name a second path with against: "fusion hit@5 must be at least the best single lane's hit@5", judged on the interval only when the two intervals do not overlap, so a new baseline never invalidates a hard-coded number. Write the rules before the run and commit them. The instrument judges a small report; edit the thresholds and watch the two columns disagree.

14Goldens: parity across implementations

rankscope goldens writes reference vectors, one line per case, in the form kind args -> answer. Another implementation prints the same lines with its own answers and rankscope goldens --check diffs them numerically with a tolerance of 1e-9. This page carries a JavaScript port of the kernel; the box below is that port computing the lines live. If the numbers here ever disagree with the file the package writes, one of the two is wrong.

15Putting it together on the command line

# 1. fit weights and floors on a development set
rankscope calibrate --lane bm25.run --lane dense.run --qrels qrels-dev.txt --gate-on dense --depth 100 --conservative --out thresholds.json

# 2. report on the test set, strata and the blind class named
rankscope evaluate --lane bm25.run --lane dense.run --qrels qrels-test.txt --thresholds thresholds.json \
    --strata strata.tsv --blind no-overlap --format markdown --out report.json

# 3. judge against rules you committed before running
rankscope verdict --report report.json --gates gates.json

# 4. is fusion better than the lexical lane alone?
rankscope compare --lane bm25.run --lane dense.run --qrels qrels-test.txt --candidate rrf --metric ndcg@10 --min-delta 0.02

# 5. in CI: did anything regress beyond noise since the stored baseline?
rankscope diff --before baseline.json --after report.json --fail-on-regression

# variants and conditions: a per-group cap, document-first ordering, a mutual-nearest-neighbour gate
rankscope evaluate --lane bm25.run --lane dense.run --qrels qrels-test.txt --group-sep '#' --cap 5 --document-first --mutual 3

# lanes and qrels may also be .json in the {query: {doc: score}} shape pytrec_eval and BEIR use;
# --format latex pastes into a paper, --per-query rows.csv opens in a spreadsheet

Order matters: calibrate on one file, evaluate on another, judge against a file that predates both. --depth cuts every lane to the same depth before anything is computed, and --conservative makes the FPIR floor a claim you can defend. The report's first number to read is not a metric. It is the share of positives your new lane surfaces that no existing lane does, and right after it, the count of judged queries missing from any lane.

16Caps and document-first re-ranking

A catalogue with sixty near-identical pages floods the top of a fused list, because its pages share vocabulary and style. A per-group cap keeps at most N hits per group and stops the flood. But a cap applied across groups also removes the right item when the right group is the flooder: a long manual ranks six of its own pages above its one right page, and a cap of five cuts the right page out. Document-first ordering decides the group first, keeps the top hit's whole group in its original order, and caps only the others. rankscope adds both as variants beside the ranking they came from, so the report shows what each rule costs and buys at doc and at group level.