# Retrieval eval, 2.1.0 ranking vs 2.0.0 ranking (resource_priors, stem_match, question_pool_floor off)
# 342 queries: git-mined sets over Civitas (76), PoliternalSite (80), PoliternalParkour (65), TerraForge (17), terra-incognita (80),
# plus 24 hand-written natural-language questions over terra-incognita (2026-09-24-terra-nl-queries.yml). tests/eval/harness.py, 2000 resamples.

| label | recall@5 | recall@10 | MRR | oracle | eff | nDCG@10 | hit@3 | P@5 | MAP | useful@budget | tokens | dup% | p50_ms | p95_ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| old | 0.541 | 0.635 | 0.521 | 0.868 | 0.600 | 0.502 | 0.599 | 0.169 | 0.426 | 0.557 | 966.8 | 0.0 | 50.5 | 92.3 |
| new | 0.557 (+0.016) | 0.653 (+0.019) | 0.531 (+0.010) | 0.877 (+0.009) | 0.606 (+0.006) | 0.513 (+0.011) | 0.605 (+0.006) | 0.173 (+0.004) | 0.434 (+0.007) | 0.579 (+0.022) | 951.5 (-15.4) | 0.0 | 51.4 (+0.9) | 101.8 (+9.5) |

paired comparison: new vs old (n=342)
| metric        |    delta |              95% CI |      p | sig |
|---|---|---|---|---|
| recall@5      |  +0.0156 | [+0.0012,+0.0327] |  0.044 | yes |
| recall@10     |  +0.0185 | [+0.0039,+0.0356] |  0.017 | yes |
| MRR           |  +0.0104 | [+0.0011,+0.0208] |  0.031 | yes |
| nDCG@10       |  +0.0110 | [+0.0032,+0.0200] |  0.005 | yes |
| hit@3         |  +0.0058 | [-0.0088,+0.0205] |  0.697 | no  |
| P@5           |  +0.0041 | [+0.0000,+0.0088] |  0.131 | no  |
| MAP           |  +0.0075 | [+0.0004,+0.0150] |  0.041 | yes |
| useful@budget |  +0.0224 | [+0.0066,+0.0404] |  0.008 | yes |
| oracle        |  +0.0088 | [+0.0000,+0.0205] |  0.247 | no  |
| cand_recall   |  +0.0073 | [+0.0000,+0.0175] |  0.247 | no  |

| label | recall@5 | recall@10 | MRR | oracle | eff | nDCG@10 | hit@3 | P@5 | MAP | useful@budget | tokens | dup% | p50_ms | p95_ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Civitas:old | 0.452 | 0.567 | 0.524 | 0.868 | 0.604 | 0.475 | 0.566 | 0.147 | 0.409 | 0.502 | 937.4 | 0.0 | 39.3 | 112.9 |
| Civitas:new | 0.455 | 0.583 | 0.545 | 0.868 | 0.628 | 0.492 | 0.579 | 0.150 | 0.424 | 0.519 | 920.2 | 0.0 | 40.9 | 112.9 |
| label | recall@5 | recall@10 | MRR | oracle | eff | nDCG@10 | hit@3 | P@5 | MAP | useful@budget | tokens | dup% | p50_ms | p95_ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PoliternalSite:old | 0.498 | 0.585 | 0.529 | 0.825 | 0.641 | 0.492 | 0.600 | 0.179 | 0.425 | 0.562 | 977.6 | 0.0 | 55.0 | 81.7 |
| PoliternalSite:new | 0.502 | 0.582 | 0.528 | 0.825 | 0.639 | 0.490 | 0.588 | 0.182 | 0.423 | 0.567 | 947.2 | 0.0 | 52.5 | 79.0 |
| label | recall@5 | recall@10 | MRR | oracle | eff | nDCG@10 | hit@3 | P@5 | MAP | useful@budget | tokens | dup% | p50_ms | p95_ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PoliternalPark:old | 0.700 | 0.796 | 0.486 | 0.892 | 0.545 | 0.557 | 0.585 | 0.147 | 0.478 | 0.669 | 898.1 | 0.0 | 43.7 | 83.7 |
| PoliternalPark:new | 0.700 | 0.796 | 0.485 | 0.892 | 0.543 | 0.556 | 0.569 | 0.147 | 0.476 | 0.669 | 897.9 | 0.0 | 49.3 | 92.3 |
| label | recall@5 | recall@10 | MRR | oracle | eff | nDCG@10 | hit@3 | P@5 | MAP | useful@budget | tokens | dup% | p50_ms | p95_ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TerraForge:old | 0.750 | 0.770 | 0.662 | 1.000 | 0.662 | 0.657 | 0.824 | 0.306 | 0.580 | 0.578 | 978.9 | 0.0 | 71.1 | 98.2 |
| TerraForge:new | 0.750 | 0.799 | 0.680 | 1.000 | 0.680 | 0.677 | 0.824 | 0.306 | 0.594 | 0.608 | 978.9 | 0.0 | 45.9 | 79.6 |
| label | recall@5 | recall@10 | MRR | oracle | eff | nDCG@10 | hit@3 | P@5 | MAP | useful@budget | tokens | dup% | p50_ms | p95_ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| terra-incognit:old | 0.457 | 0.566 | 0.535 | 0.875 | 0.612 | 0.462 | 0.613 | 0.180 | 0.369 | 0.486 | 1014.2 | 0.0 | 57.5 | 91.5 |
| terra-incognit:new | 0.479 | 0.589 | 0.545 | 0.887 | 0.614 | 0.470 | 0.625 | 0.185 | 0.372 | 0.506 | 986.7 | 0.0 | 57.9 | 115.4 |
| label | recall@5 | recall@10 | MRR | oracle | eff | nDCG@10 | hit@3 | P@5 | MAP | useful@budget | tokens | dup% | p50_ms | p95_ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| terra-nl:old | 0.667 | 0.708 | 0.428 | 0.833 | 0.514 | 0.496 | 0.542 | 0.133 | 0.428 | 0.625 | 1043.6 | 0.0 | 51.3 | 83.9 |
| terra-nl:new | 0.792 | 0.833 | 0.476 | 0.917 | 0.519 | 0.564 | 0.625 | 0.158 | 0.476 | 0.792 | 1073.0 | 0.0 | 69.5 | 138.1 |
