boost / docs / mcp-hub · RAG Phases 1–3

RAG · shipped in PR #23 · #30 · #64

# Semantic skill search, surfaced as a hub.

boost turns tapped skill repos into a searchable corpus and lets an agent ask for "the best skill for <task>." Three phases got it there: Phase 1 — full-content BM25 retrieval with an LLM rerank; Phase 2 — an opt-in dense embeddings backend; Phase 3 — an extensible MCP registry that exposes all of it. Everything degrades gracefully and the default install stays zero-dependency.

3 RAG phases shipped
6 MCP tools on the registry
89.6%Mutation kill on core/
0 Deps in the default install
The whole stack

## Three phases, one degrade-safe path

Each phase is a layer that improves retrieval without ever being required. If the dense backend isn't installed, search uses BM25; if there's no index yet, it uses frontmatter; if there's no LLM, it skips the rerank. Nothing hard-fails.

Fig 1 — The layers are a preference order, not a pipeline: retrieval picks the highest layer that's actually available and falls through on absence.

Indexing · `boost reindex`

## From cloned repos to two stores

Tapping a repo already clones its skills to local disk, so building the index is a local batch job — no extra network for BM25. Bodies are stripped of frontmatter, chunked, and written to a persisted, commit-keyed index; the opt-in dense store additionally embeds each chunk.

Fig 2 — `boost reindex` builds the BM25 store; `boost reindex --dense` additionally embeds each chunk. Both key on the git commit already recorded per tap, so only changed taps are re-read.

Query · `rag.search()`

## Retrieve, then rerank

A search is two stages. First `_retrieve_any` pulls a wide shortlist (`max(60, limit×4)`) from the best available engine; then, if an LLM is present, it reorders the top of that shortlist by task fit — otherwise it keeps the retrieval order.

Fig 3 — `search()` returns `None` only when no index exists, which is the single signal the MCP `boost_search` tool uses to fall back to frontmatter search — so a brand-new install still answers.

The hub · architecture

## One registry, many handlers

The JSON-RPC server never knows what tools exist — it asks the `Registry` in `core/mcp.py`. Handlers that reach out to the world (like GitHub) probe for their dependency and degrade to a message instead of crashing.

Fig 4 — The server holds no tool knowledge; it delegates every `tools/list` / `tools/call` to the registry. New tools slot in on the right without touching the loop on the left. `boost_search` is the handler that calls `rag.search()` from Fig 3.

The hub · sequence

## A `boost_discover_github` call, end to end

The happy path reaches GitHub; the shaded branch is what happens when `gh` is missing or the search fails — the server answers, never dies.

Fig 5 — Steps 5–6 are the only ones that touch the network. Everything below step 4 has a mirror in the shaded `alt` branch, so a missing dependency is a message, not an exception.

The refactor

## From an if/elif ladder to a registry

Same behaviour, same wire protocol — but adding tool #7 no longer means editing the dispatcher, and an unknown tool resolves in exactly one place.

before — configuration.py

    # flat dispatch: every tool is a branch
    def _mcp_tool(tool, args):
        if tool == "boost_search":
            ...
        if tool == "boost_list":
            ...
        # +1 tool = +1 branch here
        return None, False

    _MCP_TOOLS = [ {…5 hand-kept specs…} ]

after — core/mcp.py + configuration.py

    # the seam: an ordered name → (spec, handler) map
    REGISTRY = mcp.Registry()

    REGISTRY.register(
        "boost_discover_github",
        "Discover new SKILL.md repos…",
        {"type": "object", "properties": {…}},
        _tool_discover_github)          # ← the only edit

    _MCP_TOOLS = REGISTRY.specs()       # list stays in sync

Cheat sheet

## Explain it two ways

Every concept, twice: the simple version on the left, the deep technical on the right. Open both and compare — they describe the same thing at different altitudes.

🧸 Simple
🔬 Deep technical
Retrieval engine · Phases 1–2
Cut pages into index cards chunking

A whole skill page is too big to match on well. So we cut it into small overlapping index cards. Overlap means a sentence that lands on the seam still shows up whole on one card, so we never lose it.

Passage-level chunking rag.chunk()

`read_body` pulls each item's Markdown from `~/.boost/repos` and strips frontmatter; `chunk()` slices it into ~`1000`-char windows with `150`-char overlap, capped at `40` chunks per item. Retrieval is therefore passage-level: a match on one section surfaces the whole skill, scored by its best chunk.

Which words really matter BM25

To rank cards, we count shared words — but rare words ("jest") count more than common ones ("the"), and a long card doesn't win just for being long. That's the always-on ranker; it needs no internet and no AI.

Full-content BM25 k1=1.2 · b=0.75

Phase 1: a pure-stdlib sparse ranker over full bodies (`K1=1.2`, `B=0.75`), scoring by term frequency × inverse document frequency with length normalization. Per `(name, tap)` the max chunk score wins. The postings live in a persisted `rag_index.json` with the tuning `params` stored inline, so they're tunable without a format change.

Big handful, then a wise sort two-stage

First grab a generous handful of maybe-matches fast. Then, if a smart helper (an AI) is around, ask it to put the very best on top for your exact task. If it's not around, the fast ranking is already pretty good — we just use that.

Retrieve → LLM rerank rag.search()

Stage 1: `_retrieve_any` returns a wide shortlist of `max(60, limit×4)`. Stage 2: `rerank()` asks the `core/ai.py` bridge (local `claude` CLI → `ANTHROPIC_API_KEY`) to reorder names best-first, tagging the result `"Claude relevance"`; with no LLM it returns the retrieval order tagged `"BM25 full-content"`.

Search by meaning, not words dense

An optional upgrade turns every skill into a point in "meaning space," so "test my React app" can find a `jest` skill even with no shared words — it just finds the nearest points. It's optional because it needs an extra install and a key.

Dense embeddings + sqlite-vec Phase 2 · [rag]

Chunks are embedded via `embed` over urllib — preferring `VOYAGE_API_KEY` (`voyage-3`, 1024-d) then `OPENAI_API_KEY` (`text-embedding-3-small`, 1536-d) — and stored in a `sqlite-vec` `vec0(… distance_metric=cosine)` table. Retrieval does a KNN over-fetch (`max(k×8, 200)`) then reduces per-entry. A provider/dimension change invalidates the store so stale vectors are never served.

Only re-read what changed incremental

When you refresh, we don't re-read every shelf — only the ones whose books actually changed since last time. That keeps refreshes quick even with a big library.

Commit-keyed reuse boost reindex

Both stores key on the git commit already recorded per tap. On rebuild, unchanged taps are reused verbatim and only changed taps' rows are replaced (`reused` / `reindexed` in the stats). Bumping `INDEX_VERSION` wipes and rebuilds. Because taps are already cloned to disk, BM25 indexing needs no network.

The hub · Phase 3
The toy box registry

Imagine a toy box with labelled slots. Each toy knows its own name and what it does, and it puts itself into a slot. When someone asks "what toys do you have?" you just read the labels. When they say "play with the drum," you grab the drum from its slot. You never keep a giant list in your head — the box does.

`Registry`: ordered name → (spec, handler) core/mcp.py

`Registry` holds three parallel structures keyed by tool name: an insertion-ordered name list, a `specs` map (`{name, description, inputSchema}`), and a `handlers` map. `register()` rejects empty or duplicate names; `specs()` returns the JSON payload in registration order; `call(name, args)` looks up the handler and returns `(None, False)` for anything unknown. It's pure, stdlib-only, and mutation-gated.

What each toy promises handler

Every toy promises the same thing: "give me your ask, and I'll hand back some words, plus a thumbs-up or thumbs-down." That's it. Because they all promise the exact same shape, the toy box doesn't care which toy it grabbed — they're interchangeable.

The handler contract (text, is_error)

A handler is `fn(args: dict) -> (text, is_error)`. `text` is the human-readable result; `is_error` becomes the MCP `result.isError` flag. A handler may return `text is None` to mean "no result," which the server renders identically to an unknown tool — so the `-32602` path lives in exactly one place. Handlers close over the core engine (`rag`, `catalog`, `store`) but the registry knows nothing about them.

The scout that looks outside boost_discover_github

Most toys play with things already in the room. This one is a scout: it opens the door and looks down the street (GitHub) for new skills to bring home. If the door is locked (no `gh` tool installed), it doesn't panic — it just says "hey, you need a key, here's how to get one" and comes back inside.

First reach-out tool gh code search

`_tool_discover_github` probes `shutil.which("gh")`, then calls `discovery.github_skill_search(query, limit)` — a single `gh api search/code?q=filename:SKILL.md…` page (no cache write, limit clamped to 100, query URL-encoded and appended to the filename filter). Results map to `{repo, path, url, description}` and render one per line. Every failure mode (`TimeoutExpired`, `OSError`, non-zero exit, bad JSON) returns `None` so the handler emits a hint instead of raising.

Never cry, always answer degradation

The rule for any toy that reaches outside: never throw a tantrum. If something it needs is missing, it must still say something friendly and useful. That way the whole toy box keeps working even when one toy can't reach its toy.

Probe-then-degrade core/ai.py pattern

Mirrors the established `core/ai.py` discipline: detect the dependency, and if absent return a short actionable message rather than raising. The MCP server is a long-lived stdio loop — an uncaught exception in one `tools/call` would be caught by the server's blanket handler, but degrading inside the tool gives a precise, testable message and keeps `is_error` semantically correct.

Old handles still turn shims

We rearranged the room, but we kept the old light switches on the wall so anything that used to flip them still works. Nobody has to relearn where the switches are.

Back-compat: `_MCP_TOOLS` / `_mcp_tool` no churn

`_MCP_TOOLS = REGISTRY.specs()` and a one-line `_mcp_tool()` that delegates to `REGISTRY.call()` preserve the two symbols the server loop and the existing test suite reference. The refactor is behaviour-preserving: the `tools/list` order is unchanged for the first five tools, and only the new sixth entry is appended — so `TestMcp` needed a single one-line assertion update.

Extension contract

## Adding tool #7

The whole point of the seam: a new capability — a database query, another external API — is a handler plus one `register()`. No server edit, no dispatcher edit.

your_new_tool.py

    def _tool_x(args):                       # fn(args) -> (text, is_error)
        if not _dependency_available():
            return "needs <dep> — here's how to get it", True   # degrade
        return _do_the_work(args), False

    REGISTRY.register(
        "boost_x",
        "one-line description shown to the agent",
        {"type": "object", "properties": {…}},
        _tool_x)                              # tools/list + tools/call: automatic

Rule | Why
--- | ---
Handler returns `(text, is_error)` | Uniform shape lets the registry stay tool-agnostic.
Reach-out tools must degrade, never raise | The stdio server is long-lived; a missing `gh`, offline network, or absent `[rag]` extra must not kill it.
Registration order = `tools/list` order | Deterministic surface; existing clients see a stable ordering with new tools appended.
Put reusable logic in `core/` | It gets unit-tested and mutation-gated — the registry itself lives in `core/mcp.py` for exactly this reason.