boost grades its own retrieval quality.

Search quality is a number, not a vibe.

boost has several ways to find a skill — a frontmatter heuristic, a full-content BM25 index, an opt-in dense-vector backend, and an LLM reranker. Which one actually returns the right skill for a real question? The eval system answers that empirically, so a change that quietly degrades retrieval can't ship.

One golden set, four engines, five verdicts

Every tier reuses the same golden judgments and the same retrieval stack. Data flows left to right; the golden set feeds the scoring stage from above.

The required gate (Tier 1) is pure-stdlib and offline. Everything green-lit beyond it is opt-in.

Five tiers, one rule: never block on a maybe

Only Tier 1 gates the build. It is deterministic and dependency-free precisely so it can be required without ever flaking. Everything that needs a network, a key, or a heavy library is opt-in and skips cleanly when its prerequisites are absent.

What the numbers actually say

Two findings that only appeared once the instrument could see them. Both are reproducible: scripts/eval_retrieval.py --build -k 10 over the pinned corpus.

1 · The score was mostly about the corpus

The gate reported near-perfect retrieval for a long time. Growing the pinned corpus from six registries to twenty — same golden set, same queries, nothing else changed — moved every metric:

Three of the four floors in force before that change fail outright on the larger corpus. A floor calibrated where BM25 scores 1.000 is a floor about the corpus, not about retrieval. hit@1 0.473 is the honest headline: at twenty taps, fewer than half of these queries put the right item first. That is not a regression — it is what users already had, now visible.

2 · The score was also about the shape of the question

The keyword golden set grades catalog items by name, so its queries carry the vocabulary of the thing they are looking for. That is BM25's home ground. The 50 queries in golden-natural.jsonl describe a problem instead — each written from its target's own description, with the target's distinctive name tokens deliberately stripped — and over identical data BM25 scores recall 0.690 / hit@1 0.240 against 1.000 / 0.780 on the name-shaped set.

Same index, same corpus, same engine. The gap is entirely the phrasing of the question, which is why one query set cannot floor a retrieval system on its own.

3 · Which engine actually wins

The hybrid path fuses BM25 and dense by reciprocal rank rather than preferring either. That was a design decision, so here it is measured — every engine over the same corpus, both query sets, k=10. Voyage is absent: it needs an API key, and these runs were keyless, which is the configuration most users are in.

Keyword golden set — 91 queries that grade items by name:

Natural-language set — 50 queries describing a problem, name tokens stripped:

Fusing beats choosing. Hybrid wins or ties on both sets, and on the natural-language queries it beats both of its own components on every metric — which is the case for rank fusion rather than picking a favourite engine. The two components fail in opposite directions: BM25 takes hit@1 on name-shaped queries (0.791 vs 0.780), dense takes it decisively on human-phrased ones (0.420 vs 0.340). Preferring either one alone would hand half the queries to the engine that is worse at them.

Note also how far catalog.search — frontmatter only, no body index — falls on real questions: hit@1 0.080. Indexing the body is most of what makes search work, and a name-graded query set alone would never have shown it.

4 · Embedding locally is slower than the plan assumed

Building the keyless dense store over this corpus — 743 entries, 3,740 chunks, ONNX bge-small-en-v1.5 on CPU — took 4,431 s (74 minutes), about 1.2 s per chunk. Extrapolated to a full catalogue of ~28k items that is on the order of days, not the “hour-plus” the roadmap estimated. It does not weaken the keyless tier — queries embed in milliseconds and the store is built once — but it does mean shipping prebuilt per-registry vectors is a requirement rather than an optimisation.

What this changed

Both findings came from making the instrument sharper before trusting it: the gate now floors four metrics rather than recall@k alone, baselines are keyed to the query set that produced them, and the corpus is a realistic size. Every retrieval change since is reported against both query shapes, and several claims did not survive that — unchunking the index, for one, turned out to trade a little name-query recall for a large gain on natural-language queries, rather than being the free win it was written up as.

How Tier 1 runs on every PR

The golden set grades items by name, so the repos that hold them must be present. CI taps a pinned corpus first, builds the BM25 index, scores the golden set, and floors recall — all offline after the tap, all deterministic.

On the pinned corpus BM25 recall is 1.000, so the 0.85 floor leaves wide margin — a drift-resistant gate that still catches a broken index or a scoring regression.

Simple on the left, technical on the right

Same concept, two depths. Open the left card for the plain-English version, the right card for what actually happens in the code.

One command per tier

# Tier 1 — required, deterministic, part of `make check`
$ make eval
  GATE recall@10 = 1.000  (min 0.850)  -> PASS

# Tier 1b — is the difference statistically significant? (needs [eval])
$ make eval-stats
  best engine: BM25 full-content
  vs catalog.search  ndcg@10  p=0.1127  not significant  W/T/L=7/34/2

# Tier 2a / 2b — LLM rerank lift & recommendation grounding (key-gated)
$ make eval-ai   # rerank: hit@1 +0.163 over raw BM25
$ make eval-rec  # recommend: grounded 1.000, 0 hallucinated picks

# Tier 2c — explain faithfulness via ragas (needs [eval] + a judge key)
$ make eval-explain
  skill                         faithfulness
  pdf                                  0.94
  MEAN                                 0.91   (n=7)

Illustrative output. Opt-in tiers print a one-line hint and exit 0 when a dependency or key is missing — they never block.