Metadata-Version: 2.5
Name: laghu
Version: 0.3.1
Summary: model2vec static models in a fraction of the memory, with the same vectors
Project-URL: Repository, https://github.com/vedicreader/laghu
Project-URL: Documentation, https://vedicreader.github.io/laghu/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: embeddings,mmap,model2vec,nbdev,static-embeddings,tokenizers,unigram
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: fastcore>=1.8
Requires-Dist: huggingface-hub>=0.24
Requires-Dist: marisa-trie>=1.2
Requires-Dist: numpy>=1.24
Requires-Dist: tokenizers>=0.20
Provides-Extra: dev
Requires-Dist: ipykernel>=7.3.0; extra == 'dev'
Requires-Dist: model2vec<0.10,>=0.9; extra == 'dev'
Requires-Dist: nbdev>=3.3.14; extra == 'dev'
Requires-Dist: notebook>=7.6.3; extra == 'dev'
Provides-Extra: fallback
Requires-Dist: model2vec<0.10,>=0.9; extra == 'fallback'
Provides-Extra: rust
Requires-Dist: laghu-rs>=0.1.0; (python_version >= '3.11') and extra == 'rust'
Description-Content-Type: text/markdown

# laghu


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

[model2vec](https://github.com/MinishLab/model2vec) makes static embedding models: models that turn a text into a vector by looking up a stored vector for each of its tokens and averaging them. There is no neural network to run, so they are very fast, and the whole model is two files: an embedding table with one row per token, and a tokenizer that splits text into those tokens. laghu (लघु, Sanskrit for “light”) loads the same models and gives the same vectors with far less memory. A model loaded with laghu has model2vec’s `encode`, `encode_as_sequence` and `tokenize`, and its output matches model2vec’s bit for bit. The one difference, for an empty list of texts, is listed at the end of this page.

The memory goes on two things. model2vec reads the whole embedding table into each process; for potion-multilingual-128M that is 512 MB. That model’s tokenizer is a Unigram tokenizer, which splits text by choosing the most likely sequence of pieces from a list of 500,353, and the `tokenizers` library keeps a search structure (a character trie) over those pieces that takes about another 610 MB. After encoding, a model2vec process holds about 1.3 GB of private memory, memory that belongs to that process alone and cannot be shared.

laghu avoids both copies. It maps the table from `model.safetensors` into memory read-only, so the operating system loads pages of the file only when they are read, keeps them in its page cache, shares them between every process that maps the same file, and can drop them again when memory is short. For a Unigram model it also replaces the tokenizer’s trie: it builds a vocabulary file once per tokenizer.json, holding the pieces in a compact [marisa trie](https://github.com/pytries/marisa-trie) with their ids and scores, which every process maps and shares in the same way, and it segments text with a Viterbi search written in Python, or in Rust when the optional `laghu-rs` wheel is installed. The text normalization and the first split into words still come from `tokenizers`, built from the same tokenizer.json with the vocabulary left out. The same model then takes about 60 MB of private memory per process, most of it the Python interpreter and its imports.

Models whose tokenizer is WordPiece, BPE or word-level keep using `tokenizers` for tokenizing. laghu still maps their tables, so how much they save depends on how large the table is compared with the tokenizer.

## Install

``` sh
uv add laghu        # or: pip install laghu
```

model2vec is not a dependency. Install the extra, `uv add 'laghu[fallback]'`, if you want [`load`](https://vedicreader.github.io/laghu/encoder.html#load) to fall back to model2vec when laghu cannot read a model, or want to use [`same_as_model2vec`](https://vedicreader.github.io/laghu/encoder.html#same_as_model2vec). The extra installs model2vec 0.9, which laghu matches. model2vec 0.10 gives different vectors, and its `tokenize` no longer takes `max_length`.

laghu has an optional Rust core, the wheel `laghu-rs` (module `laghu_rs`, Python 3.11 or later). Install it with `pip install laghu-rs`, or with laghu’s extra, `uv add 'laghu[rust]'`, from laghu 0.3.0.

## The optional Rust core

With `laghu-rs` installed, laghu tokenizes Unigram models in Rust, on all cores, with the GIL released. `laghu_rs` normalizes, splits out added tokens and pre-tokenizes with the `tokenizers` crate, then runs laghu’s Viterbi search over a byte trie in the shared vocabulary file. For a float32 table without weights or a token mapping, it also averages the vectors. The ids and vectors are the same as the pure Python path’s and model2vec’s, bit for bit.

potion-multilingual-128M encodes 500-character chunks at 39,750 chunks/s with `laghu-rs` on the 14 cores of an M3 Max. The same model encodes 1,736 chunks/s in pure Python and 5,276 chunks/s with model2vec 0.9. The process holds 64 MB of private memory after encoding with `laghu-rs`.

laghu tokenizes in pure Python when any of these is true:

- `laghu_rs` cannot be imported.
- `LAGHU_RUST=0` is set when the model is loaded.
- The model is loaded with `shared=False`, or its tokenizer.json truncates or pads.

laghu never uses `laghu-rs` for a model it tokenizes with `tokenizers`: WordPiece, BPE and WordLevel models, and Unigram models that use a feature laghu does not reproduce.

laghu splits text in Python and segments it in Rust when any of these is true:

- The installed Python `tokenizers` has a different major or minor version from `laghu_rs.TOKENIZERS`, the `tokenizers` crate version `laghu-rs` was built with. A different patch version is accepted. `laghu-rs` 0.1.0 is built with 0.23.2 and accepts any Python `tokenizers` 0.23.x. The `PROBE` check below still applies.
- The `tokenizers` crate cannot read the tokenizer.json.
- The crate splits any text in `PROBE`, 46 of laghu’s built-in difficult texts, differently from laghu’s Python split. laghu logs a warning.

The ids are the same on every path.

## Use

``` python
from laghu import load

m = load('minishlab/potion-multilingual-128M')      # a folder, or a Hugging Face repo id (the local cache is tried first)
v = m.encode(['The quick brown fox.', 'कर्मण्येवाधिकारस्ते'])   # float32, (2, 256), unit rows: the same array model2vec returns
m.tokenize(['hello world'])
m.encode_as_sequence('hello world').shape          # (n_tokens, 256)
```

    (2, 256)

[`load`](https://vedicreader.github.io/laghu/encoder.html#load) takes a local folder or a Hugging Face repo id, and returns a [`LeanModel`](https://vedicreader.github.io/laghu/encoder.html#leanmodel). A repo already in the local Hugging Face cache loads without the network. If laghu cannot load the model, [`load`](https://vedicreader.github.io/laghu/encoder.html#load) logs why and tries model2vec’s `StaticModel` instead, which needs `laghu[fallback]`; if that fails too, it raises laghu’s original error. Pass `fallback=False` to get the error without trying model2vec. Neither loader can read tables stored as bfloat16 or float8, because numpy has no such types.

The vocabulary file for a Unigram model is kept in `$LAGHU_CACHE`, else `$XDG_CACHE_HOME/laghu`, else `~/.cache/laghu`, and next to tokenizer.json when the cache cannot be written. The first load builds it in a second or two; later loads, in any process, map it in a few hundredths of a second. If tokenizer.json changes or the file is damaged, it is built again. `load(..., shared=False)` keeps the vocabulary in a Python dict instead, which costs about 80 MB more per process.

To check a model of your own, `same_as_model2vec(path)` loads it both ways and compares the results on laghu’s built-in test texts, or on texts you pass:

``` python
from laghu import same_as_model2vec
same_as_model2vec('minishlab/potion-base-8M', ['my', 'own texts'])
```

    /Users/71293/code/personal/orgs/laghu/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm

    {'lean': False,
     'n': 2,
     'ids': True,
     'dtype': True,
     'f32': True,
     'f16': True,
     'exact': True,
     'seq': True,
     'median': True,
     'unk': True,
     'tokens': True}

The result says, for each check, whether the two loaders agreed: the token ids (`ids`), the vectors cast to float32 (`f32`) and to float16 (`f16`), the exact dtype (`dtype`) and bits (`exact`), `encode_as_sequence` (`seq`), `median_token_length` (`median`), the unknown token id (`unk`) and the token list (`tokens`). `lean` says whether laghu used its own Unigram tokenizer, and `n` is the number of texts.

## What matches, and what it costs

Every model below gives model2vec’s ids and vectors, bit for bit, on 4,726 texts. The texts include prose in 36 languages and random strings from 30 Unicode blocks, and also control characters, special tokens, emoji sequences, combining marks, bidirectional text controls and a 200,000-character string. Each memory figure comes from a fresh process. On macOS it is `phys_footprint` after encoding, and `lifetime_max_phys_footprint` for the peak. On Linux the same columns would hold `RssAnon` and the peak resident memory `VmHWM`. Speed is `encode` over 3,226 texts on an M3 Max. The “laghu tokenizes with” column says whether laghu used its own Viterbi search, in Python or in Rust with `laghu-rs`, or `tokenizers`. The first two values in the private MB and texts/s columns were measured without `laghu-rs`. The third value was measured with `laghu-rs` active, in the same way. `laghu-rs` changes only how laghu tokenizes Unigram models. Rows tokenized with `tokenizers`, including every WordPiece, BPE and WordLevel row, show `—` for it.

Of the Unigram rows, only potion-multilingual-128M has been measured with `laghu-rs`. It was measured on 2026-10-23 as the best of three runs, with laghu using Rust for both the split and the search. The machine was loaded: the load average was 6.5 over one minute and about 20 over fifteen. In the same session model2vec encoded 22,846 texts/s and laghu without `laghu-rs` 14,859 texts/s, against 39,105 and 10,902 in the older columns. Compare the `laghu-rs` figure with those same-session figures, not with the older columns. The Rust runs each took 0.025 to 0.077 seconds, and the slowest gave 41,988 texts/s.

| model | tokenizer | laghu tokenizes with | vocab × dims | dtype | norm | identical | private MB, model2vec → laghu → laghu-rs | peak MB, model2vec → laghu | texts/s, model2vec → laghu → laghu-rs |
|----|----|----|----|----|----|----|----|----|----|
| potion-base-2M | WordPiece | tokenizers | 29,528 × 64 | float32 | on | yes | 81 → 58 → — | 82 → 58 | 84,510 → 92,982 → — |
| potion-base-4M | WordPiece | tokenizers | 29,528 × 128 | float32 | on | yes | 94 → 62 → — | 95 → 62 | 87,248 → 90,242 → — |
| potion-base-8M | WordPiece | tokenizers | 29,528 × 256 | float32 | on | yes | 114 → 70 → — | 115 → 70 | 79,415 → 81,810 → — |
| potion-base-32M | WordPiece | tokenizers | 63,091 × 512 | float32 | on | yes | 244 → 97 → — | 244 → 97 | 75,009 → 81,379 → — |
| potion-retrieval-32M | WordPiece | tokenizers | 63,091 × 512 | float32 | on | yes | 244 → 97 → — | 244 → 97 | 70,799 → 79,717 → — |
| potion-science-8M | WordPiece | tokenizers | 29,528 × 256 | float32 | on | yes | 113 → 70 → — | 114 → 70 | 81,008 → 84,532 → — |
| potion-code-16M | WordPiece | tokenizers | 61,826 × 256 | float32 +map +weights | on | yes | 147 → 110 → — | 211 → 117 | 10,674 → 10,425 → — |
| potion-code-16M-v2 | WordPiece | tokenizers | 63,457 × 256 | float16 | on | yes | 108 → 101 → — | 186 → 119 | 21,819 → 22,164 → — |
| potion-multilingual-128M | Unigram | Python Viterbi, or Rust with laghu-rs | 500,353 × 256 | float32 | on | yes | 1,348 → 61 → 59 | 1,348 → 61 | 39,105 → 10,902 → 130,007 |
| M2V_base_output | WordPiece | tokenizers | 29,528 × 256 | float32 | off | yes | 112 → 69 → — | 113 → 69 | 65,460 → 87,100 → — |
| M2V_multilingual_output | WordPiece | tokenizers | 501,054 × 256 | float32 | off | yes | 834 → 198 → — | 842 → 295 | 42,565 → 90,283 → — |
| M2V_base_glove | WordLevel | tokenizers | 400,002 × 256 | float32 | off | yes | 656 → 186 → — | 656 → 194 | 67,028 → 85,058 → — |
| M2V_base_glove_subword | WordPiece | tokenizers | 400,779 × 256 | float32 | off | yes | 647 → 178 → — | 649 → 195 | 66,256 → 85,430 → — |
| jina-embeddings-v3-separation-distilled | Unigram | Python Viterbi, or Rust with laghu-rs | 250,002 × 256 | float32 | off | yes | 722 → 88 → not measured | 722 → 88 | 51,836 → 20,840 → not measured |
| multilingual-e5-large-m2v | Unigram | Python Viterbi, or Rust with laghu-rs | 250,002 × 512 | float32 | on | yes | 981 → 109 → not measured | 981 → 109 | 24,139 → 19,670 → not measured |
| m2v-gemma-embedding-300m | Unigram | Python Viterbi, or Rust with laghu-rs | 255,732 × 256 | float16 +weights | on | yes | 542 → 93 → not measured | 542 → 93 | 20,319 → 12,134 → not measured |
| potion-multilingual-128M-i8-tokenade | Unigram | Python Viterbi, or Rust with laghu-rs | 500,353 × 256 | int8 | on | yes | 985 → 138 → not measured | 985 → 138 | 27,676 → 12,090 → not measured |
| m2v-qwen3-embedding-0.6b-1024d | Unigram | Python Viterbi, or Rust with laghu-rs | 151,644 × 1024 | float16 +weights | on | yes | 602 → 132 → not measured | 602 → 132 | 16,792 → 7,168 → not measured |
| ovos-m2v-intents-es-ES-bne-v5 | Unigram | Python Viterbi, or Rust with laghu-rs | 50,259 × 64 | float32 | on | yes | 169 → 52 → not measured | 169 → 52 | 11,786 → 11,265 → not measured |
| m2v-gte-256-edu | Unigram | tokenizers | 313,845 × 256 | float32 | on | yes | 1,028 → 663 → — | 1,028 → 675 | 25,363 → 24,981 → — |
| bge-m3-m2v-256 | Unigram | Python Viterbi, or Rust with laghu-rs | 250,002 × 256 | float32 | off | yes | 725 → 89 → not measured | 725 → 89 | 34,518 → 20,670 → not measured |
| potion-mxbai-micro | WordPiece | tokenizers | 2,000 × 256 | int8 +map +weights | on | yes | 90 → 72 → — | 91 → 72 | 68,385 → 70,998 → — |
| NIFE-mxbai-embed-large-v1_model2vec | WordPiece | tokenizers | 107,962 × 1024 | float32 | off | yes | 558 → 106 → — | 572 → 116 | 41,042 → 26,756 → — |
| ovos-m2v-intents-pt-PT-albertina-v5 | BPE | tokenizers | 50,262 × 256 | float32 | on | yes | 162 → 90 → — | 162 → 90 | 46,343 → 22,813 → — |
| F2LLM-v2-330M-model2vec | BPE | tokenizers | 151,644 × 512 | float16 | on | yes | 356 → 171 → — | 356 → 171 | 28,795 → 40,132 → — |
| static-retrieval-multilingual-69m-v1 | BPE | tokenizers | 179,936 × 384 | float32 | off | yes | 537 → 229 → — | 541 → 263 | 32,235 → 31,848 → — |

`+map` marks a vocabulary-quantized model, which stores fewer rows than it has tokens and a mapping from each token to its row. `+weights` marks a model that scales each token’s vector by a stored weight. The potion-multilingual-128M row was measured again on 2026-10-13 with the shared vocabulary file. The other Unigram rows were measured with the vocabulary in a dict, which cost potion-multilingual-128M about 80 MB more private memory per process.

Without `laghu-rs`, laghu is slower for Unigram models, at between about a third of model2vec’s rate and about the same. `tokenizers` spreads a batch over all cores, while the Python Viterbi runs on one. With `laghu-rs`, laghu also runs on all cores and is faster than model2vec on potion-multilingual-128M: 39,750 against 5,276 chunks/s, as measured in [the optional Rust core](#the-optional-rust-core). The other Unigram models have not been measured with `laghu-rs`. These are batch measurements and say nothing about the time to encode a single query.

laghu keeps a cache of words it has already segmented. It helps when the tokenizer first splits the text into words, as jina-v3, bge-m3 and multilingual-e5 do: natural text runs 1.6 to 1.9 times faster on a first pass and 3 to 5 times faster when the same text comes again. The [bench](https://vedicreader.github.io/laghu/bench.html) page records these measurements and why two faster Rust tokenizers, tokie and gigatoken, were not adopted.

For the other models both loaders tokenize with the same `tokenizers` code. Their timing samples last about 0.1 seconds and vary by up to half between runs, so the differences in speed there mean nothing.

## What is not supported

- Quantizing or truncating a model while loading it (model2vec’s `quantize_to`, `dimensionality` and `vocabulary_quantization`). Each would make a new copy of the table, which is what laghu exists to avoid. Load with model2vec if you need them.
- A Unigram model with `single_word` added tokens is tokenized with `tokenizers`, though its table is still mapped. `tokenizers` decides where a word ends with Unicode’s Alphabetic property, and Python’s `str.isalpha` does not match it.
- `use_multiprocessing` is accepted and ignored; the result is the same.
- `encode([])` returns an empty array of shape `(0, dim)`, where model2vec raises.

## Developing

laghu uses nbdev 3 with hatchling: the notebooks in `nbs/` are the source. `uv run nbdev-prepare` exports the package, runs the tests and rebuilds this README. `uv run nbdev-test --flags slow` also runs the benchmark, which downloads about 5 GB of models.

## Licence

MIT.
