Metadata-Version: 2.4
Name: thinletter-eq
Version: 0.1.0
Summary: Measure a compressed query encoder against your own document index: import, synthetic queries, fp32 cache, llama.cpp compile, paired evaluation.
Author-email: thinletter <info@thinletter.io>
License-Expression: Apache-2.0
Project-URL: Homepage, https://thinletter.io
Project-URL: Repository, https://github.com/rosecky/embedding-quantization-public
Project-URL: Report, https://github.com/rosecky/embedding-quantization-public/blob/main/docs/release/technical_report.md
Project-URL: Models, https://huggingface.co/thinletter
Keywords: embeddings,retrieval,quantization,llama.cpp,gguf,query-encoder,evaluation
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Intended Audience :: Developers
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: numpy>=1.24
Provides-Extra: torch
Requires-Dist: torch>=2.1; extra == "torch"
Requires-Dist: transformers>=4.40; extra == "torch"
Requires-Dist: sentence-transformers>=3.0; extra == "torch"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Dynamic: license-file

# thinletter-eq

Measure a **compressed query encoder** against **your own document index** — the index stays exactly as it is, only
the query side gets smaller. One command line, no repository checkout: import your corpus and its existing vectors,
generate synthetic queries, compile a client with llama.cpp, evaluate it against the unchanged index with a paired
interval, and read the verdict of a release rule we use ourselves.

Method and numbers: [technical report](https://github.com/rosecky/embedding-quantization-public/blob/main/docs/release/technical_report.md)
· released clients: [huggingface.co/thinletter](https://huggingface.co/thinletter) · browser runtime for the
vector-quantised files: [`@thinletterio/vqweb`](https://www.npmjs.com/package/@thinletterio/vqweb) · Apache-2.0.

## Install

```
pip install thinletter-eq              # numpy only: import, compile, eval, compare, report
pip install "thinletter-eq[torch]"     # + torch, transformers, sentence-transformers: synth, cache
```

`eq compile` and `eq eval --clients` call the llama.cpp binaries (`llama-imatrix`, `llama-quantize`, `llama-embedding`,
build b81c99b or later): put them on `PATH` or pass `--llama-bin <dir>` / `--bin <file>`. `eq doctor` tells you what
is missing for which step.

## The recipe

```
eq import  --name mycorpus --docs corpus.jsonl --doc-emb index.f32 --doc-ids ids.json --teacher qwen3-0.6b --language cs
eq synth   --dataset mycorpus --lang Czech --n-per-doc 2 --as-calib          # queries + calibration text, no real queries needed
eq cache   --dataset mycorpus --teacher qwen3-0.6b                           # fp32 query vectors of the unchanged model
eq compile --model qwen3-embedding-0.6b-f16.gguf --types Q4_K_M Q3_K --calib data/calib/mycorpus_synth_q.txt --table q4_0
eq eval    --dataset mycorpus --teacher qwen3-0.6b --clients models/*.gguf --bin llama-embedding --rule
eq compare results/perq/mycorpus_<a>.npz results/perq/mycorpus_<b>.npz     # paired bootstrap over the same queries
eq report  --out clients.md
```

`corpus.jsonl` has one `{"id", "title", "text"}` per line; `index.f32` (or `.npy`) holds your document vectors in the row
order of `ids.json`. They are never re-encoded: they *are* the index the client is measured against. With query
vectors too (`--queries`, `--query-emb`) the fp32 model's top-10 is written as pseudo-relevance, labelled as such — then
nDCG@10 means "agreement with the fp32 ranking". Without judgements the harness reports cosine to the fp32 query vector
and top-10 overlap, which is what an unlabelled corpus can support.

A client that runs elsewhere (the WebGPU runtime, a service): `eq queries --dataset mycorpus --texts` prints what to
encode, in order; `eq eval --client-emb vectors.npy --client-name myclient` scores the result like any other client.

## What the numbers mean

`eq eval` reports nDCG@10 and recall@100 (NaN without judgements), cosine to the fp32 query vector and top-10 overlap
with the fp32 ranking, and writes one per-query file per client. `eq compare` draws a paired bootstrap interval over
queries (10 000 draws, seed 0) between two such files: same queries, one thing changed. Read it against two variances
the interval does not contain — the calibration draw (about 0.01 nDCG@10 between two equally good quantisations) and
the machine (about 0.006) — so differences under 0.01 are ties.

Release rule (`--rule`): keep the smallest client with nDCG@10 at least 95 % of fp32, the interval of the difference
above −0.02, cosine at least 0.94 and top-10 overlap at least 0.75. What we found with it (report §2–3): at 3.4 bits
a 0.6B encoder keeps 96–101 % of its retrieval quality on four English corpora; at low bit widths the *language* of the
calibration text matters more than its domain; tolerance to compression is a property of the checkpoint (one 0.6B model
holds 99 % at 3.4 bits, another of the same architecture fails below 4.5); llama.cpp's own quantiser with an importance
matrix is as good as anything fancier at 200 MiB, which is why `eq compile` is exactly that.

Teachers known to the package (`--teacher`): `qwen3-0.6b` (Qwen/Qwen3-Embedding-0.6B), `harrier-0.6b`
(microsoft/harrier-oss-v1-0.6b), `bge-m3` (BAAI/bge-m3), `me5-small`, `qwen3-4b`, `e5-mistral-7b`, and the other keys
of `eq.teacher.TEACHERS`; each carries the query prompt, pooling and context length the evaluation must reproduce.

## Not in this package

The vector-quantised pipeline (2–3.6 bits per weight, the `.vqw` container, GPTQ onto the K-quant grid) lives in the
[repository](https://github.com/rosecky/embedding-quantization-public) as scripts; it pulls in research modules that
are not ready to be a library. Its released files run in the browser through `@thinletterio/vqweb`, and `eq eval
--client-emb` measures them like any other client.

## Licence, contact

Apache-2.0. Jan Rosecký, thinletter — info@thinletter.io.
