boost / docs / evaluation system

Quality · shipped in PR #112 · #142 · #145 · #147

# boost grades its own retrieval quality.

A package manager for AI skills is only as good as the skill it hands back for "the best thing for <task>." So boost measures that — with a required, deterministic golden-set gate that fails the build on a retrieval regression, plus opt-in tiers for statistical significance and LLM-judged faithfulness. Everything degrades gracefully; the required gate has zero dependencies.

5 Evaluation tiers
0.919 BM25 recall@10 · full catalog
43 Golden judgments
0 Deps in the required gate
The problem

## Search quality is a number, not a vibe.

boost has several ways to find a skill — a frontmatter heuristic, a full-content BM25 index, an opt-in dense-vector backend, and an LLM reranker. Which one actually returns the right skill for a real question? The eval system answers that empirically, so a change that quietly degrades retrieval can't ship.

The core idea is the classic information-retrieval one: a golden set of `query → the skill(s) that should come back` judgments, scored with standard IR metrics (recall@k, hit@1, MRR, nDCG). It's the same shape as a TREC `qrels` file — hand-labelled ground truth that turns "seems good" into "recall went from 0.92 to 0.71, block the merge."

Architecture

## One golden set, four engines, five verdicts

Every tier reuses the same golden judgments and the same retrieval stack. Data flows left to right; the golden set feeds the scoring stage from above.

The required gate (Tier 1) is pure-stdlib and offline. Everything green-lit beyond it is opt-in.

The contract

## Five tiers, one rule: never block on a maybe

Only Tier 1 gates the build. It is deterministic and dependency-free precisely so it can be required without ever flaking. Everything that needs a network, a key, or a heavy library is opt-in and skips cleanly when its prerequisites are absent.

Tier | What it grades | Metric / gate | Needs |
--- | --- | --- | --- | ---
1 · retrieval | Does the right skill come back for a real query? | `recall@k ≥ 0.85` floor vs the pinned corpus | nothing — pure stdlib, offline | required
1b · significance | Is engine A really better than B, or just lucky? | ranx paired Student's t-test (`p<0.05`) | `[eval]` extra (ranx) | opt-in
2a · rerank | How much does the LLM reranker improve ordering? | hit@1 / MRR / nDCG lift over raw BM25 | claude CLI or `ANTHROPIC_API_KEY` | opt-in
2b · recommend | Are `boost recommend`'s AI picks relevant and grounded? | precision@k + hard grounding gate (0 hallucinated picks) | claude CLI or `ANTHROPIC_API_KEY` | opt-in
2c · explain | Does `boost explain` stay true to the SKILL.md? | ragas `faithfulness` (claims verified vs source) | `[eval]` extra + judge key | opt-in

Sequence · the required gate

## How Tier 1 runs on every PR

The golden set grades items by name, so the repos that hold them must be present. CI taps a pinned corpus first, builds the BM25 index, scores the golden set, and floors recall — all offline after the tap, all deterministic.

On the pinned corpus BM25 recall is 1.000, so the 0.85 floor leaves wide margin — a drift-resistant gate that still catches a broken index or a scoring regression.

Explained twice

## Simple on the left, technical on the right

Same concept, two depths. Open the left card for the plain-English version, the right card for what actually happens in the code.

◍ SIMPLE
◍ TECHNICAL
# The golden set
What is a "golden set"?

A hand-written answer key. Each row is a question a user might type and the skill that should come back. To grade any search engine, you run every question through it and check whether the right answer showed up near the top.

qrels + IR metrics

`tests/eval/golden.jsonl` holds `{query, relevant[], kind}` rows — a TREC-style qrels file. The harness runs each ranker, then computes `recall@k`, `hit@1`, `MRR` and `nDCG@k` with binary relevance, sliced per item kind (skill / rule / workflow).

# Why BM25 beats the heuristic
Why not just match keywords?

The simple approach only reads a skill's short description. BM25 reads the whole skill file, so it finds matches the description never mentions. The eval proves it: BM25 finds the right skill far more often.

0.756 → 0.919 recall@10

The frontmatter heuristic (`catalog.search`) scores recall@10 = 0.756; full-content BM25 (`rag.retrieve`) scores 0.919 on the same 43 queries. That measured +0.163 is the evidence that justifies the RAG stack's complexity — not a hunch.

# Is the win real? (Tier 1b)
Could a better score be luck?

On only a few dozen questions, one engine can look better just by chance. A significance test asks: if you re-ran on fresh questions, would the winner keep winning? It reports "real difference" or "too close to call."

ranx paired t-test

`--stats` feeds the per-query scores to `ranx`, which runs a paired Student's t-test between engines and reports a p-value plus win/tie/loss. Deterministic, offline. On the small pinned corpus the gap is not yet significant (p≈0.11–0.26) — an honest signal to widen the set.

# Faithful explanations (Tier 2c)
Can `explain` make things up?

`boost explain` writes a plain-English summary of a skill. The risk is it invents abilities the skill doesn't have. This tier checks every sentence against the real skill file and scores how much of the summary is actually supported.

ragas faithfulness

`scripts/eval_explain.py` generates the explanation with boost's own `ai.ask`, then scores it with ragas `faithfulness`: the answer is decomposed into claims and each is verified against the SKILL.md context. LLM-judged, so it's key-gated and never part of `make check`.

# Why a pinned corpus
Where do the skills come from?

To grade "did the right skill come back," the skills have to exist. CI starts empty, so the gate first downloads a fixed, hand-picked list of skill repos — always the same list, so the score means the same thing every run.

taps.txt + ensure_eval_corpus.sh

`tests/eval/taps.txt` is the minimal repo set covering all 44 golden targets (`boost tap --defaults` omits every rule/workflow repo). `ensure_eval_corpus.sh` taps it into `$BOOST_HOME` — idempotent via a sentinel. Regression-vs-baseline is relaxed (`--regression-eps 1`) so upstream drift can't flake the required gate.

Run it yourself

## One command per tier

make · boost evaluation tiers

    # Tier 1 — required, deterministic, part of `make check`
    $ make eval
      GATE recall@10 = 1.000  (min 0.850)  -> PASS

    # Tier 1b — is the difference statistically significant? (needs [eval])
    $ make eval-stats
      best engine: BM25 full-content
      vs catalog.search  ndcg@10  p=0.1127  not significant  W/T/L=7/34/2

    # Tier 2a / 2b — LLM rerank lift & recommendation grounding (key-gated)
    $ make eval-ai   # rerank: hit@1 +0.163 over raw BM25
    $ make eval-rec  # recommend: grounded 1.000, 0 hallucinated picks

    # Tier 2c — explain faithfulness via ragas (needs [eval] + a judge key)
    $ make eval-explain
      skill                         faithfulness
      pdf                                  0.94
      MEAN                                 0.91   (n=7)

Illustrative output. Opt-in tiers print a one-line hint and exit 0 when a dependency or key is missing — they never block.