Metadata-Version: 2.4
Name: static-site-search-eval
Version: 0.1.0
Summary: Chunk, embed, and evaluate semantic search over a static site's markdown
Project-URL: Homepage, https://github.com/bartdegoede/static-site-search-eval
Project-URL: Issues, https://github.com/bartdegoede/static-site-search-eval/issues
Author: Bart de Goede
License-Expression: MIT
License-File: LICENSE
Keywords: embeddings,evaluation,model2vec,search,static-site
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.12
Requires-Dist: model2vec>=0.6.0
Requires-Dist: numpy>=2.1
Requires-Dist: pyyaml>=6.0
Description-Content-Type: text/markdown

# static-site-search-eval

`sss-eval` chunks a static site's markdown, embeds the chunks with a static embedding
model (model2vec/potion), writes browser-readable index artifacts, and evaluates
retrieval quality against a query set. The point of the package is the claim in its
name: the whole thing was built to find out whether a static site can ship real
semantic search without shipping a neural network to the browser. It can.

## Why the model fits in a few megabytes

A model2vec / potion model is not a neural network. It is a lookup table: one
row of floats per vocabulary token. There is no attention, no matrix multiply
chain, no ONNX runtime, no WASM binary. Inference is four steps:

```python
ids = tokenizer.encode(text)          # WordPiece token ids, no [CLS]/[SEP]
rows = embedding[ids]                 # gather: one row per token id
pooled = rows.mean(axis=0)            # mean-pool the *unnormalized* rows
vector = pooled / np.linalg.norm(pooled)  # L2-normalize the pooled result
```

That's the forward pass. Training (distilling a real sentence-transformer down to
a static table via PCA) is expensive; using the result is a gather and a mean.
Because inference has no learned computation graph, the entire "model" a browser
needs is the token table itself — an `int8` matrix plus a small vocabulary list —
which is why the winning configuration below downloads 4.21 MB instead of the
23+ MB an ONNX sentence-transformer needs for the same job.

## Install

```bash
pip install static-site-search-eval
# or
uv add static-site-search-eval
```

Requires Python 3.12+.

## Building an index

The winning configuration found by the eval in this repository (see
[Results](#results) below):

```bash
sss-eval build \
  --corpus content/post \
  --outdir static/search \
  --model minishlab/potion-base-8M \
  --dims 128 \
  --chunk-size 600 \
  --chunk-overlap 120
```

Title-prefixing (each chunk's embedded text is prefixed with its post title) is
the default; pass `--no-title-prefix` to disable it. `--cache-root` (default
`.embed-cache`) controls where content-hash-keyed embeddings are cached between
runs, so re-running `build` after editing one post only re-embeds that post's
chunks.

This writes six files to `--outdir`:

- `manifest.json`
- `chunks.<hash>.json`
- `docs.<hash>.bin`
- `tokens.<hash>.bin`
- `scales.<hash>.bin`
- `vocab.<hash>.json`

and prints a one-line summary:

```
313 chunks, 128d, vocab 29528 -> static/search
```

To confirm the index only points at anchors that actually exist in the rendered
site (heading anchors are indexed today so that section-level deep links can be
turned on later without re-embedding, but nothing renders them yet — data that
nothing reads rots quietly unless something checks it):

```bash
hugo
sss-eval verify-anchors --search-dir static/search --public public
```

## The artifact format

A browser only ever fetches `manifest.json` by a fixed URL; everything else it
names is content-hashed (`<stem>.<sha256-prefix-12>.<ext>`) and can be cached
forever. Bump the corpus or the model and the hash changes, so there is no
versioning scheme to get wrong and no stale-cache class of bug — `manifest.json`
is the only response that must be served with `no-cache`.

| file | contents |
|---|---|
| `manifest.json` | model id, dims, chunk size/overlap, `doc_scale` (127.0), and the filenames of the other five artifacts. The only URL that must never be cached. |
| `chunks.<hash>.json` | JSON array, one record per row of `docs.bin`, same order: `{post, title, href, snippet, heading, anchor}`. |
| `docs.<hash>.bin` | raw `int8`, shape `n_chunks × dims`, C order (row-major, row `i` is chunk `i`'s vector). Quantized with a single **global** scale of 127 because document vectors are already L2-normalized. |
| `tokens.<hash>.bin` | raw `int8`, shape `vocab_size × dims`, C order. This *is* the model — the entire embedding table, quantized. |
| `scales.<hash>.bin` | `float32`, little-endian, length `vocab_size` — one scale **per row** of `tokens.bin`. Token rows carry the model's zipf/SIF-style downweighting in their magnitude (that's how it downweights "the" without a stopword list), so a single global scale would destroy that signal; each row needs its own. |
| `vocab.<hash>.json` | JSON array of token strings, indexed by token id — the id `tokens.bin` row `i` corresponds to is the position of that id's string in this array. |

Every binary artifact is a raw `ArrayBuffer`, never base64. Base64 costs +33%
on the largest file shipped (`tokens.bin`); for a payload whose entire point is
its download size, that is not a rounding error.

## The three tokenizer traps

Reimplementing potion's tokenizer in JavaScript (rather than shipping a WASM
tokenizer, which would defeat the purpose) means reproducing three
non-obvious behaviors of HuggingFace `tokenizers`, each verified against
`model2vec`'s own `StaticModel.tokenize()`:

1. **No `[CLS]` / `[SEP]`.** `tokenize()` calls
   `tokenizer.encode_batch_fast(sentences, add_special_tokens=False)`. The
   `TemplateProcessing` post-processor recorded in `tokenizer.json` describes
   how special tokens *would* be inserted — it is a decoy left over from the
   tokenizer's BERT ancestry. `add_special_tokens=False` means it never runs.
   A browser implementation that adds `[CLS]`/`[SEP]` ids will embed a vector
   for a token sequence the Python pipeline never produces.

2. **`[UNK]` ids are deleted, not embedded.** After tokenizing, model2vec
   filters `unk_token_id` out of the id list entirely:
   `[t for t in token_ids if t != unk_token_id]`. It does not gather the
   `[UNK]` row and mean-pool it in. A query that is entirely out-of-vocabulary
   (e.g. a string of emoji, or a language the tokenizer's vocab doesn't cover)
   tokenizes to an **empty** id list, and mean-pooling zero rows must yield a
   zero vector rather than throwing or returning the `[UNK]` embedding.

3. **Accents are stripped even though the config says they aren't.**
   `tokenizer.json`'s normalizer sets `"strip_accents": null`. That reads like
   "leave accents alone." It doesn't: HuggingFace `tokenizers`' `BertNormalizer`
   treats a `null` `strip_accents` as "inherit from `lowercase`," and
   `lowercase` is `true`. So accents ARE stripped — `café` and `cafe` tokenize
   identically. A browser reimplementation that takes the JSON at face value
   and skips accent-stripping will silently diverge from the Python tokenizer
   on any query containing a diacritic.

## Two facts a browser implementer needs

Both measured against this package's own quantization code
(`src/sss_eval/quantize.py`), not assumed:

1. **The browser never has to dequantize `docs.bin`.** Cosine similarity is
   invariant to a positive per-row scale, and every row of `docs.bin` was
   quantized with the *same* global scale (127, applied to unit vectors), so
   that scale is a constant factor across every document's score — it cannot
   change the ranking. A `Float32Array` query vector dotted directly against
   the raw `int8` document rows produces the same ranking as dotting against
   the dequantized `float32` rows, to within `1e-6`. Query-side, the query
   vector is built once per search from the float32 token table math above,
   dequantizing only what's needed (the tokens actually present in the
   query), never the 313-row-or-larger document matrix.

2. **`int8 @ int8` wraps around silently.** A 300-dimension dot product of two
   int8 vectors whose true mathematical value is 3,000,000 does not throw, does
   not produce `NaN`, and does not saturate — it wraps modulo 256 per
   multiply-accumulate step in a naively-typed accumulator and can return
   something like `-64`. There is no exception to catch. The fix is
   structural, not defensive: accumulate the running dot product into a
   `Float32Array` (or a plain JS number, which is already a float64), never
   into an `Int8Array` or `Int32Array` sized to the inputs.

## Reproducing the eval

The eval this package shipped is pinned against a frozen corpus snapshot; see
[`examples/degoe-de/SNAPSHOT.md`](examples/degoe-de/SNAPSHOT.md) for exactly
which commit of which site it was taken from and why it is not meant to be
refreshed — the numbers in the [Results](#results) section below and in the
accompanying blog post are only meaningful against that exact snapshot.

```bash
uv sync
pnpm install

# Dump chunk texts (and query texts) to JSON so the Node arms embed exactly
# what Python did -- same chunker, same text, different runtime.
uv run python -m sss_eval.dump_chunks \
  --corpus examples/degoe-de/corpus \
  --out build/chunks.json \
  --queries examples/degoe-de/queries.yaml

# Embed those chunks and queries with the non-potion arms, each with the
# real runtime a browser would use: transformers.js (ONNX q8) for MiniLM,
# the ternlight WASM engine for its two arms. build_arms (below) looks for
# these three files next to --outdir, i.e. in --outdir's parent.
node node/build_minilm.mjs build/chunks.json build/minilm.json
node node/build_ternlight.mjs build/chunks.json build/ternlight-base.json @ternlight/base
node node/build_ternlight.mjs build/chunks.json build/ternlight-mini.json @ternlight/mini

# Rank the same 30 queries with the site's real Fuse.js config, for the
# keyword baseline and the RRF-fused hybrid arms.
node node/rank_fuse.mjs examples/degoe-de/index.json examples/degoe-de/queries.yaml build/fuse-ranks.json

# Assemble every arm (potion swept over dims/chunking/title-prefix, plus the
# Node arms above) into the format evaluate.py reads.
uv run python -m sss_eval.build_arms \
  --corpus examples/degoe-de/corpus \
  --queries examples/degoe-de/queries.yaml \
  --manifest build/arms.json

# Score every arm, semantic-only and RRF-fused with the keyword baseline,
# against the 30-query eval set, and apply the pre-registered ship rule
# (smallest download within 0.03 recall@3 of the best arm; MRR breaks ties).
uv run python -m sss_eval.evaluate \
  --queries examples/degoe-de/queries.yaml \
  --arms build/arms.json \
  --keyword-ranks build/fuse-ranks.json \
  --json build/results.json
```

Node is only needed to reproduce the MiniLM/ternlight/keyword arms of the
eval; it is not a dependency of `sss-eval build` or of using the artifacts in
a browser.

## Results

Winner: **`potion-base-8M`, PCA-truncated to 128 dims, 600-char chunks with 120
overlap, title-prefixed**, RRF-fused with a keyword ranking (Fuse.js, already
shipped with the site).

| | download | recall@1 | recall@3 | MRR@10 |
|---|---|---|---|---|
| potion-base-8M (128d, 600/120, title-prefixed) + RRF | 4.21 MB | 0.717 | 0.967 | 0.944 |
| MiniLM-L6-v2, ONNX q8 | 23.10 MB | 0.700 | — | — |

First-query download of 4,205,974 bytes (4.21 MB) is the `int8` token table
plus its `float32` per-row scales plus the JSON vocabulary — the three files a
browser needs before it can embed a single query. The document index itself,
for the 13-post corpus in `examples/degoe-de/`, is 313 chunks × 128 `int8`
dims = 40 KB, fetched once and cached alongside the rest of the site's static
assets.

<!-- TODO: link the blog post once published -->

## License

The code in this repository is MIT licensed (see [`LICENSE`](LICENSE)).
`examples/degoe-de/corpus/` contains Bart de Goede's blog posts, included as
evaluation data for the eval above — that content is copyright Bart de Goede
and is not covered by the MIT license.
