Metadata-Version: 2.4
Name: tobler-search
Version: 0.1.0
Summary: Query-adaptive context assembly for code retrieval: the score distribution decides how much context a query gets, within a token budget.
Project-URL: Repository, https://github.com/datacompose/tobler
Project-URL: Issues, https://github.com/datacompose/tobler/issues
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: fastmcp>=3.4.4
Requires-Dist: numpy>=2.5.1
Requires-Dist: semble==0.3.4
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/logo.svg" width="150" alt="tobler: isoline contours around a query point">
</p>

# tobler

**Query-adaptive context assembly for code retrieval.** Tobler's first law for your context window: near things are more related than distant things, so let the score distribution decide how much context a query deserves, instead of a fixed top-5.

If you drive an AI coding agent, tobler feeds it the right code in fewer tokens, and on large repos it finds the target chunk where a fixed top-5 misses it two-thirds of the time. It sits on top of [semble](https://pypi.org/project/semble/) retrieval and ships as a library and a FastMCP server.

**Requirements:** Python ≥ 3.12 and an MCP client (e.g. Claude Code). `semble` is pulled in automatically (pinned). Published on PyPI as [`tobler-search`](https://pypi.org/project/tobler-search/) (the import name stays `tobler`).

## What it does in practice

- Sharp query ("what does BACKOFF_BASE_MS do") → N_eff = 3, a few full bodies + a map, ε honest.
- Diffuse query ("how does real-time collaboration work") → N_eff = 550, wide map of signatures and stubs.
- Same token budget both times. Fixed top-k is wrong in both directions on the same repo.

## Benchmark: hit rate (ground-truth symbol queries, ~40 per repo)

τ was calibrated once on two repos and never touched again; the first three rows are repos it never saw:

| repo                                    | semble top-5 | tobler B=1000 | tobler B=2000 |
| --------------------------------------- | ------------ | ------------- | ------------- |
| **numpy** (16,465 chunks)               | 32%          | 88%           | 92%           |
| **ripgrep** (Rust)                      | 48%          | 92%           | 95%           |
| **zod** (TypeScript)                    | 80%          | 95%           | 98%           |
| 8 smaller repos (247 queries)           | 76%          | 99%           | 100%          |
| calibration repos (80 q, _not_ held out) | 94%          | 98%           | 99%           |

Fixed top-5 degrades hard as repos grow: on numpy it finds the right chunk a third of the time. Tobler holds ≥88% everywhere with zero per-repo tuning.

## Benchmark: tokens (mean per query)

| repo            | semble top-5 | tobler B=1000 | tobler B=2000 |
| --------------- | ------------ | ------------- | ------------- |
| numpy           | 1463         | 995           | 1940          |
| ripgrep         | 1544         | 993           | 1944          |
| zod             | 1453         | 981           | 1940          |
| 8 smaller repos | 1395         | 992           | 1966          |

Semble's spend is uncontrolled (whatever five chunks happen to cost, observed range ~1,000–2,800/query); tobler's budget is a hard ceiling that was never exceeded on any of ~370 benchmark queries. **Tokens per correct answer** (mean tokens ÷ hit rate): on numpy semble costs ~4,570 tokens per hit vs tobler's ~1,130 at B=1000, so it is **4× more token-efficient**; ~3× on ripgrep, ~1.8× on small repos. When the budget genuinely can't cover the relevant code, ε reports it instead of silently truncating.

## The math (seven equations, one calibrated knob)

1. **Center/standardize** raw cosine scores over all N chunks: `s̃ = (s − s̄)/σ`, which removes the "everything resembles everything" anisotropy baseline.
2. **Softmax at temperature τ**: `ŵᵢ = softmax(s̃/τ)`.
3. **Effective count** `N_eff = 1/Σŵᵢ²`: how many chunks genuinely matter for this query.
4. **Nucleus cut**: smallest set S with mass ≥ p (default 0.9), so k adapts per query.
5. **Residual tail** `ε = 1 − Σ_S ŵᵢ`, returned to the caller so the agent knows when it's reasoning inside a truncation.
6. **Budget allocation** `bᵢ = B·ŵᵢ/Σ_S ŵⱼ`, mapped to level-of-detail tiers (full body → signature+docstring → path+symbol), with greedy repair rolling unspent budget down by weight.
7. **Session smoothing**: `m_t = λ_t·m_{t−1} + (1−λ_t)·s̃_t` with `λ_t = λ_max·max(0, corr(s̃_t, m_{t−1}))`. An anaphoric follow-up ("what are the arguments to that") gets pulled toward the session's focus; a topic pivot has corr ≈ 0, so memory vanishes with no reset heuristic. The gate experiment calibrated λ_max* = 1.0. The knob removed itself: the correlation *is* the blend. Follow-up hit rate 9% → 46%, pivots unharmed at 96% (see `experiments/RESULT.md` Addendum 3).

τ is the one *calibrated* free parameter (found by maximizing var(log N_eff) across real queries); p, k_min, and the budget B are fixed defaults you can override. On two real repos, τ* = 1.28 on the hybrid channel and N_eff spans two orders of magnitude (p90/p10 = 215×); see `experiments/RESULT.md`.

## Why the scores are hybrid

Scores come from a hybrid channel (z-scored model2vec cosine blended with z-scored raw BM25) because static embedders can't see rare identifiers (`mulberry32` ranks ~200th semantically, 3rd lexically). A k_min=5 floor on the nucleus cut guarantees coverage never drops below the old fixed top-5. Warm-index latency: numpy 54 ms/query, ripgrep 15 ms, zod 14 ms.

## Quickstart

Install:

```bash
pip install tobler-search        # or: uv pip install tobler-search
```

Register the MCP server with Claude Code (uvx needs no separate install):

```bash
claude mcp add --scope user tobler -- uvx --from tobler-search tobler-mcp
```

Tool: `search(query, repo, budget=2000, content="code", fresh=False)` → markdown with an honesty header (`N_eff=… k=… ε=… tokens=…/…`, plus `mem=…` when session memory shaped the result) followed by tiered context blocks. Related follow-up queries are smoothed toward the session's focus; pass `fresh=true` to drop the memory for a query.

Run the server directly (stdio) with `tobler-mcp`.

## CLI

`pip install tobler-search` puts the `tobler` command on your PATH. For shells, scripts, or agents without MCP access, the same search runs from the command line:

```bash
tobler index ./repo                          # warm the semble index first (optional)
tobler search "authentication flow" ./repo   # print the adaptive context
tobler search "save model to disk" ./repo --budget 3000
tobler search "deployment guide" ./repo --content docs
```

Output is the same honesty header + tiered blocks the MCP tool returns. (Session smoothing is MCP-only; the CLI is stateless per call.)

## Agent skill

`skills/tobler-search/` ships an agent skill that teaches a coding agent when and how to reach for tobler (MCP first, CLI fallback) and how to read the `N_eff / k / ε / mem` header. Point your agent's skill loader at it, or copy it into your skills directory.

## Reproduce the benchmarks

From a source checkout (`git clone https://github.com/datacompose/tobler`):

```bash
uv run pytest                                      # 322 tests
uv run experiments/neff_sweep.py <repo1> <repo2>   # the τ calibration
uv run experiments/benchmark.py <repo1> [...]      # hit rate + tokens vs semble
uv run experiments/session_sweep.py <repo1> [...]  # the session-smoothing gate
```

## Layout

- `src/tobler/core.py`: the pipeline math, pure numpy
- `src/tobler/bridge.py`: the only module touching semble private attrs (pinned `semble==0.3.4`)
- `src/tobler/render.py`: tier rendering + token cost estimates
- `src/tobler/assemble.py`: steps 1–7 composed; `Session` holds the smoothing memory
- `src/tobler/mcp.py`: FastMCP server, one session stream per (repo, content)
- `src/tobler/cli.py`: the `tobler` command (`search` + `index`)
- `skills/tobler-search/`: shippable agent skill wrapping the MCP tool + CLI
- `experiments/`: the validation gates (Phase 0, hybrid benchmark, session smoothing: all passed)

## License

MIT. See [LICENSE](LICENSE).

---

Built in a night, but the numbers aren't hand-wavy: τ calibrated once on held-out repos, every claim gated by an experiment in `experiments/`. Kick the tires and file issues.
