Metadata-Version: 2.5
Name: laghu
Version: 0.2.1
Summary: model2vec static models in a fraction of the memory, with the same vectors
Project-URL: Repository, https://github.com/vedicreader/laghu
Project-URL: Documentation, https://vedicreader.github.io/laghu/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: embeddings,mmap,model2vec,nbdev,static-embeddings,tokenizers,unigram
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: fastcore>=1.8
Requires-Dist: huggingface-hub>=0.24
Requires-Dist: marisa-trie>=1.2
Requires-Dist: numpy>=1.24
Requires-Dist: tokenizers>=0.20
Provides-Extra: dev
Requires-Dist: ipykernel>=7.3.0; extra == 'dev'
Requires-Dist: model2vec<0.10,>=0.9; extra == 'dev'
Requires-Dist: nbdev>=3.3.14; extra == 'dev'
Requires-Dist: notebook>=7.6.3; extra == 'dev'
Provides-Extra: fallback
Requires-Dist: model2vec<0.10,>=0.9; extra == 'fallback'
Description-Content-Type: text/markdown

# laghu


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

[model2vec](https://github.com/MinishLab/model2vec) makes static embedding models: models that turn a text into a vector by looking up a stored vector for each of its tokens and averaging them. There is no neural network to run, so they are very fast, and the whole model is two files: an embedding table with one row per token, and a tokenizer that splits text into those tokens. laghu (लघु, Sanskrit for “light”) loads the same models and gives the same vectors with far less memory. A model loaded with laghu has model2vec’s `encode`, `encode_as_sequence` and `tokenize`, and its output matches model2vec’s bit for bit. The one difference, for an empty list of texts, is listed at the end of this page.

The memory goes on two things. model2vec reads the whole embedding table into each process; for potion-multilingual-128M that is 512 MB. That model’s tokenizer is a Unigram tokenizer, which splits text by choosing the most likely sequence of pieces from a list of 500,353, and the `tokenizers` library keeps a search structure (a character trie) over those pieces that takes about another 610 MB. After encoding, a model2vec process holds about 1.3 GB of private memory, memory that belongs to that process alone and cannot be shared.

laghu avoids both copies. It maps the table from `model.safetensors` into memory read-only, so the operating system loads pages of the file only when they are read, keeps them in its page cache, shares them between every process that maps the same file, and can drop them again when memory is short. For a Unigram model it also replaces the tokenizer’s trie: it builds a vocabulary file once per tokenizer.json, holding the pieces in a compact [marisa trie](https://github.com/pytries/marisa-trie) with their ids and scores, which every process maps and shares in the same way, and it segments text with a Viterbi search written in Python. The text normalization and the first split into words still come from `tokenizers`, built from the same tokenizer.json with the vocabulary left out. The same model then takes about 60 MB of private memory per process, most of it the Python interpreter and its imports.

Models whose tokenizer is WordPiece, BPE or word-level keep using `tokenizers` for tokenizing. laghu still maps their tables, so how much they save depends on how large the table is compared with the tokenizer.

## Install

``` sh
uv add laghu        # or: pip install laghu
```

model2vec is not a dependency. Install the extra, `uv add 'laghu[fallback]'`, if you want [`load`](https://vedicreader.github.io/laghu/encoder.html#load) to fall back to model2vec when laghu cannot read a model, or want to use [`same_as_model2vec`](https://vedicreader.github.io/laghu/encoder.html#same_as_model2vec). The extra installs model2vec 0.9, which laghu matches. model2vec 0.10 gives different vectors, and its `tokenize` no longer takes `max_length`.

## Use

``` python
from laghu import load

m = load('minishlab/potion-multilingual-128M')      # a folder, or a Hugging Face repo id (the local cache is tried first)
v = m.encode(['The quick brown fox.', 'कर्मण्येवाधिकारस्ते'])   # float32, (2, 256), unit rows: the same array model2vec returns
m.tokenize(['hello world'])
m.encode_as_sequence('hello world').shape          # (n_tokens, 256)
```

    (2, 256)

[`load`](https://vedicreader.github.io/laghu/encoder.html#load) takes a local folder or a Hugging Face repo id, and returns a [`LeanModel`](https://vedicreader.github.io/laghu/encoder.html#leanmodel). A repo already in the local Hugging Face cache loads without the network. If laghu cannot load the model, [`load`](https://vedicreader.github.io/laghu/encoder.html#load) logs why and tries model2vec’s `StaticModel` instead, which needs `laghu[fallback]`; if that fails too, it raises laghu’s original error. Pass `fallback=False` to get the error without trying model2vec. Neither loader can read tables stored as bfloat16 or float8, because numpy has no such types.

The vocabulary file for a Unigram model is kept in `$LAGHU_CACHE`, else `$XDG_CACHE_HOME/laghu`, else `~/.cache/laghu`, and next to tokenizer.json when the cache cannot be written. The first load builds it in a second or two; later loads, in any process, map it in a few hundredths of a second. If tokenizer.json changes or the file is damaged, it is built again. `load(..., shared=False)` keeps the vocabulary in a Python dict instead, which costs about 80 MB more per process.

To check a model of your own, `same_as_model2vec(path)` loads it both ways and compares the results on laghu’s built-in test texts, or on texts you pass:

``` python
from laghu import same_as_model2vec
same_as_model2vec('minishlab/potion-base-8M', ['my', 'own texts'])
```

    /Users/71293/code/personal/orgs/laghu/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm

    {'lean': False,
     'n': 2,
     'ids': True,
     'dtype': True,
     'f32': True,
     'f16': True,
     'exact': True,
     'seq': True,
     'median': True,
     'unk': True,
     'tokens': True}

The result says, for each check, whether the two loaders agreed: the token ids (`ids`), the vectors cast to float32 (`f32`) and to float16 (`f16`), the exact dtype (`dtype`) and bits (`exact`), `encode_as_sequence` (`seq`), `median_token_length` (`median`), the unknown token id (`unk`) and the token list (`tokens`). `lean` says whether laghu used its own Unigram tokenizer, and `n` is the number of texts.

## What matches, and what it costs

Every model below gives model2vec’s ids and vectors, bit for bit, on 4,726 texts. The texts include prose in 36 languages and random strings from 30 Unicode blocks, and also control characters, special tokens, emoji sequences, combining marks, bidirectional text controls and a 200,000-character string. Each memory figure comes from a fresh process. On macOS it is `phys_footprint` after encoding, and `lifetime_max_phys_footprint` for the peak. On Linux the same columns would hold `RssAnon` and the peak resident memory `VmHWM`. Speed is `encode` over 3,226 texts on an M3 Max. The “laghu tokenizes with” column says whether laghu used its Python Viterbi or `tokenizers`.

| model | tokenizer | laghu tokenizes with | vocab × dims | dtype | norm | identical | private MB, model2vec → laghu | peak MB, model2vec → laghu | texts/s, model2vec → laghu |
|----|----|----|----|----|----|----|----|----|----|
| potion-base-2M | WordPiece | tokenizers | 29,528 × 64 | float32 | on | yes | 81 → 58 | 82 → 58 | 84,510 → 92,982 |
| potion-base-4M | WordPiece | tokenizers | 29,528 × 128 | float32 | on | yes | 94 → 62 | 95 → 62 | 87,248 → 90,242 |
| potion-base-8M | WordPiece | tokenizers | 29,528 × 256 | float32 | on | yes | 114 → 70 | 115 → 70 | 79,415 → 81,810 |
| potion-base-32M | WordPiece | tokenizers | 63,091 × 512 | float32 | on | yes | 244 → 97 | 244 → 97 | 75,009 → 81,379 |
| potion-retrieval-32M | WordPiece | tokenizers | 63,091 × 512 | float32 | on | yes | 244 → 97 | 244 → 97 | 70,799 → 79,717 |
| potion-science-8M | WordPiece | tokenizers | 29,528 × 256 | float32 | on | yes | 113 → 70 | 114 → 70 | 81,008 → 84,532 |
| potion-code-16M | WordPiece | tokenizers | 61,826 × 256 | float32 +map +weights | on | yes | 147 → 110 | 211 → 117 | 10,674 → 10,425 |
| potion-code-16M-v2 | WordPiece | tokenizers | 63,457 × 256 | float16 | on | yes | 108 → 101 | 186 → 119 | 21,819 → 22,164 |
| potion-multilingual-128M | Unigram | Python Viterbi | 500,353 × 256 | float32 | on | yes | 1,348 → 61 | 1,348 → 61 | 39,105 → 10,902 |
| M2V_base_output | WordPiece | tokenizers | 29,528 × 256 | float32 | off | yes | 112 → 69 | 113 → 69 | 65,460 → 87,100 |
| M2V_multilingual_output | WordPiece | tokenizers | 501,054 × 256 | float32 | off | yes | 834 → 198 | 842 → 295 | 42,565 → 90,283 |
| M2V_base_glove | WordLevel | tokenizers | 400,002 × 256 | float32 | off | yes | 656 → 186 | 656 → 194 | 67,028 → 85,058 |
| M2V_base_glove_subword | WordPiece | tokenizers | 400,779 × 256 | float32 | off | yes | 647 → 178 | 649 → 195 | 66,256 → 85,430 |
| jina-embeddings-v3-separation-distilled | Unigram | Python Viterbi | 250,002 × 256 | float32 | off | yes | 722 → 88 | 722 → 88 | 51,836 → 20,840 |
| multilingual-e5-large-m2v | Unigram | Python Viterbi | 250,002 × 512 | float32 | on | yes | 981 → 109 | 981 → 109 | 24,139 → 19,670 |
| m2v-gemma-embedding-300m | Unigram | Python Viterbi | 255,732 × 256 | float16 +weights | on | yes | 542 → 93 | 542 → 93 | 20,319 → 12,134 |
| potion-multilingual-128M-i8-tokenade | Unigram | Python Viterbi | 500,353 × 256 | int8 | on | yes | 985 → 138 | 985 → 138 | 27,676 → 12,090 |
| m2v-qwen3-embedding-0.6b-1024d | Unigram | Python Viterbi | 151,644 × 1024 | float16 +weights | on | yes | 602 → 132 | 602 → 132 | 16,792 → 7,168 |
| ovos-m2v-intents-es-ES-bne-v5 | Unigram | Python Viterbi | 50,259 × 64 | float32 | on | yes | 169 → 52 | 169 → 52 | 11,786 → 11,265 |
| m2v-gte-256-edu | Unigram | tokenizers | 313,845 × 256 | float32 | on | yes | 1,028 → 663 | 1,028 → 675 | 25,363 → 24,981 |
| bge-m3-m2v-256 | Unigram | Python Viterbi | 250,002 × 256 | float32 | off | yes | 725 → 89 | 725 → 89 | 34,518 → 20,670 |
| potion-mxbai-micro | WordPiece | tokenizers | 2,000 × 256 | int8 +map +weights | on | yes | 90 → 72 | 91 → 72 | 68,385 → 70,998 |
| NIFE-mxbai-embed-large-v1_model2vec | WordPiece | tokenizers | 107,962 × 1024 | float32 | off | yes | 558 → 106 | 572 → 116 | 41,042 → 26,756 |
| ovos-m2v-intents-pt-PT-albertina-v5 | BPE | tokenizers | 50,262 × 256 | float32 | on | yes | 162 → 90 | 162 → 90 | 46,343 → 22,813 |
| F2LLM-v2-330M-model2vec | BPE | tokenizers | 151,644 × 512 | float16 | on | yes | 356 → 171 | 356 → 171 | 28,795 → 40,132 |
| static-retrieval-multilingual-69m-v1 | BPE | tokenizers | 179,936 × 384 | float32 | off | yes | 537 → 229 | 541 → 263 | 32,235 → 31,848 |

`+map` marks a vocabulary-quantized model, which stores fewer rows than it has tokens and a mapping from each token to its row. `+weights` marks a model that scales each token’s vector by a stored weight. The potion-multilingual-128M row was measured again on 2026-10-13 with the shared vocabulary file. The other Unigram rows were measured with the vocabulary in a dict, which cost potion-multilingual-128M about 80 MB more private memory per process.

For Unigram models laghu is slower, at between about a third of model2vec’s rate and about the same. `tokenizers` spreads a batch over all cores, while the Python Viterbi runs on one. These are batch measurements and say nothing about the time to encode a single query.

laghu keeps a cache of words it has already segmented. It helps when the tokenizer first splits the text into words, as jina-v3, bge-m3 and multilingual-e5 do: natural text runs 1.6 to 1.9 times faster on a first pass and 3 to 5 times faster when the same text comes again. The [bench](https://vedicreader.github.io/laghu/bench.html) page records these measurements and why two faster Rust tokenizers, tokie and gigatoken, were not adopted.

For the other models both loaders tokenize with the same `tokenizers` code. Their timing samples last about 0.1 seconds and vary by up to half between runs, so the differences in speed there mean nothing.

## What is not supported

- Quantizing or truncating a model while loading it (model2vec’s `quantize_to`, `dimensionality` and `vocabulary_quantization`). Each would make a new copy of the table, which is what laghu exists to avoid. Load with model2vec if you need them.
- A Unigram model with `single_word` added tokens is tokenized with `tokenizers`, though its table is still mapped. The Rust code decides where a word ends with Unicode’s Alphabetic property, and Python’s `str.isalpha` does not match it.
- `use_multiprocessing` is accepted and ignored; the result is the same.
- `encode([])` returns an empty array of shape `(0, dim)`, where model2vec raises.

## Developing

laghu uses nbdev 3 with hatchling: the notebooks in `nbs/` are the source. `uv run nbdev-prepare` exports the package, runs the tests and rebuilds this README. `uv run nbdev-test --flags slow` also runs the benchmark, which downloads about 5 GB of models.

## Licence

MIT.
