Metadata-Version: 2.5
Name: grouptok
Version: 0.3.1
Summary: A translation-grouped tokenizer: subwords that translate each other share a group
Project-URL: Homepage, https://github.com/henryhale/grouptok
Project-URL: Documentation, https://henryhale.github.io/grouptok/
Project-URL: Issues, https://github.com/henryhale/grouptok/issues
Author: Henry Hale
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: bpe,cross-lingual,multilingual,nlp,tokenizer,word alignment
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: tokenizers>=0.15
Provides-Extra: datasets
Requires-Dist: datasets>=2.14; extra == 'datasets'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=1.0; extra == 'docs'
Provides-Extra: test
Requires-Dist: pytest>=7; extra == 'test'
Provides-Extra: train
Requires-Dist: torch>=2.0; extra == 'train'
Requires-Dist: transformers>=4.38; extra == 'train'
Description-Content-Type: text/markdown

<div align=center>

# grouptok

A translation-grouped tokenizer: subwords that translate each other share a group

[![PyPI](https://img.shields.io/pypi/v/grouptok)](https://pypi.org/project/grouptok/)
[![Python](https://img.shields.io/pypi/pyversions/grouptok)](https://pypi.org/project/grouptok/)
[![Tests](https://github.com/henryhale/grouptok/actions/workflows/tests.yml/badge.svg)](https://github.com/henryhale/grouptok/actions/workflows/tests.yml)
[![License](https://img.shields.io/github/license/henryhale/grouptok)](https://github.com/henryhale/grouptok/blob/master/LICENSE)

</div>

<!-- the docs site's home page is this README from here on (docs/index.md) -->
<!-- --8<-- [start:docs] -->
**grouptok** trains a byte-level BPE tokenizer on sentence pairs. It then aligns the subwords of each pair with a neural
word aligner and puts subwords that are often aligned into one **group**. Each token becomes a `(group, member)` pair:

```
 " house" -> (412, 0)      " maison" -> (412, 1)      " Haus" -> (412, 2)
 " king"  -> (977, 0)      " roi"    -> (977, 1)
```

A language model can then predict the group first and the member second, so translations share most of their
parameters. The tokenizer itself is an ordinary BPE tokenizer (it saves a standard `tokenizer.json`, so it also loads in
`transformers`), plus a `groups.json` that holds the grouping.

## Install

```bash
pip install grouptok            # load and use a trained tokenizer (needs only `tokenizers`)
pip install "grouptok[train]"   # train one (adds torch and transformers for the aligner)
pip install "grouptok[train,datasets]"   # and train on Hugging Face datasets directly (see Sources)
```

## Training a tokenizer

Training needs the `train` extra. It runs on a GPU when one is available.

```python
from grouptok import GroupedTokenizer, TokenizerConfig

pairs = [("The king said unto the people", "Le roi dit au peuple"), ...]   # (one language, the other)
tok = GroupedTokenizer.train(pairs, TokenizerConfig(vocab_size=8192))
tok.save_pretrained("my-tokenizer")      # tokenizer.json, tokenizer_config.json, groups.json
```

## Using a trained tokenizer

Load a tokenizer saved in a local folder:

```python
from grouptok import GroupedTokenizer

tok = GroupedTokenizer.from_pretrained("my-tokenizer")
```

or one published on the Hugging Face Hub (see [Publishing a trained tokenizer](#publishing-a-trained-tokenizer)). 
`from_pretrained` reads local folders only, so download the repository first; `huggingface_hub` comes with `tokenizers`:

```python
from huggingface_hub import snapshot_download

tok = GroupedTokenizer.from_pretrained(snapshot_download("your-name/my-tokenizer"))
```

Then:

```python
ids = tok.encode("The king", bos=True, eos=True)
tok.pairs(ids)                           # [(group, member), ...] for each token
tok.decode(ids)                          # 'The king'
tok.group_of(" king")                    # [' king', ' roi']
tok.readable_groups()[:5]                # groups with at least two members, as token strings
```

The grouping is a `Grouping`, a one-to-one map between token ids and `(group, member)` pairs, which a model needs for
its embedding tables and heads:

```python
g = tok.grouping
g.token_group, g.token_member            # per-token group and member ids (tuples of length vocab_size)
g.num_groups, g.max_group_size           # sizes of the group and member tables
g.token(group, member)                   # back to the token id
```

## Publishing a trained tokenizer

Save the trained tokenizer and upload the folder to a repository on the Hugging Face Hub with the
[`hf` command](https://huggingface.co/docs/huggingface_hub/guides/cli). `hf upload` creates the repository if it doesn't
exist yet.

```bash
hf auth login                                        # once, with a token that has write access
hf upload your-name/my-tokenizer my-tokenizer        # repository id, then the folder save_pretrained wrote
```

If `hf` isn't found, update `huggingface_hub`: `pip install -U huggingface_hub`.

Anyone can then download it and load it with grouptok:

```python
from huggingface_hub import snapshot_download
from grouptok import GroupedTokenizer

tok = GroupedTokenizer.from_pretrained(snapshot_download("your-name/my-tokenizer"))
```

The repository holds a standard `tokenizer.json` and `tokenizer_config.json`, so
`transformers.AutoTokenizer.from_pretrained("your-name/my-tokenizer")` also works. 
It loads the BPE tokenizer only and ignores `groups.json`.

## More than two languages

Pairs don't have to share a language pair: pool `en–fr`, `en–de` and `fr–de` pairs and train once. 
A token joins the group of whatever it is most often aligned with, so a group can hold `" house"`, `" maison"` and `" Haus"`. 

The group size cap(`max_group_size`, default 8) bounds how many members a group can get.

```python
from grouptok import GroupedTokenizer, GroupingConfig, TokenizerConfig

pairs = en_fr_pairs + en_de_pairs + fr_de_pairs
tok = GroupedTokenizer.train(pairs, TokenizerConfig(vocab_size=32000, grouping=GroupingConfig(max_group_size=8)))
```

## Sources

Instead of loading the pairs yourself, describe each dataset as a `Source`, with its columns, row count and, if it
needs one, its own aligner, and pass them all to `train`:

```python
from grouptok import AlignerConfig, GroupedTokenizer, Source, TokenizerConfig

sources = [
    Source("Helsinki-NLP/opus-100", subset="en-fr", columns=("translation.en", "translation.fr"), rows=200_000),
    Source("michsethowusu/english-swahili_sentence-pairs", columns=("English", "Swahili"), rows=200_000),
    Source("Helsinki-NLP/opus-100", subset="en-ha", columns=("translation.en", "translation.ha"),
           aligner=AlignerConfig(model="Davlan/afro-xlmr-large")),
    Source("pairs/en-am.jsonl", columns=("en", "am")),
]
tok = GroupedTokenizer.train(sources, TokenizerConfig(vocab_size=49152, aligner=AlignerConfig(model="FacebookAI/xlm-roberta-base")))
```

- `path` is a local `.jsonl` file, or a Hugging Face dataset id (streamed; needs the `datasets` extra).
- `columns` are the two sides of the pairs, in order, and the first one supplies member 0 of the groups it creates (put
  the language every pair shares first). A dot reads a nested field. Every source names them.
- `rows` caps the rows read; rows with an empty column are skipped and don't count.
- `aligner` overrides the training aligner for this source. Pairs are aligned in one pass per aligner, and the link
  counts of all passes are pooled before grouping. Different encoders link at different rates, so a language aligned
  by a denser one wins more group slots: prefer one encoder that covers every language, and override only the
  languages it doesn't cover.

Sources are read into memory before training, like pairs.

## Configuration

```python
from grouptok import AlignerConfig, GroupingConfig, TokenizerConfig

TokenizerConfig(
    vocab_size=8192,
    special_tokens=("<|endoftext|>", "<|im_start|>", "<|im_end|>"),   # pad, bos, eos -> ids 0, 1, 2
    additional_tokens=("<think>", "</think>"),                        # more reserved tokens, never grouped
    chat_template=None,                                               # Jinja template saved to tokenizer_config.json
    grouping=GroupingConfig(max_group_size=8, min_link_count=5, min_dice=0.1),
    aligner=AlignerConfig(model="aneuraz/awesome-align-with-co", layer=8, threshold=1e-3,
                          batch_size=64, max_length=128, device=None, fp16=True),
)
```

Any Hugging Face encoder with a fast tokenizer can be the aligner ([mBERT](https://huggingface.co/google-bert/bert-base-multilingual-cased),
[XLM-R](https://huggingface.co/FacebookAI/xlm-roberta-base),
[awesome-align](https://huggingface.co/aneuraz/awesome-align-with-co), ...). You can also pass
your own object to `GroupedTokenizer.train(..., aligner=...)`. It needs an `encode(texts)` method that returns the
vectors `[batch, length, dim]` and the character spans `[batch, length, 2]` of its word pieces, with empty spans for
special and padding positions.

## Command line

```bash
grouptok train --pairs europarl.jsonl opus.jsonl:src,tgt --columns en,fr --limit 200000 --out my-tokenizer --vocab-size 8192
grouptok inspect my-tokenizer --word " king" " house"
grouptok encode my-tokenizer "The king said"
```

`--pairs` takes JSONL files whose two sides are the `--columns`; write `FILE:A,B` to choose the fields of one file.
`grouptok train --help` lists every option.

Instead of `--pairs`, `--config` takes a JSON list of [sources](#sources). A source's `aligner` overrides only the
fields it sets; the rest, and the whole aligner of sources without one, come from the aligner options (`--aligner`,
`--layer`, `--device`, ...):

```bash
grouptok train --config sources.json --out my-tokenizer --vocab-size 49152 --aligner FacebookAI/xlm-roberta-base
```

```json
[
  {"path": "Helsinki-NLP/opus-100", "subset": "en-fr", "columns": ["translation.en", "translation.fr"], "rows": 200000},
  {"path": "Helsinki-NLP/opus-100", "subset": "en-ha", "columns": ["translation.en", "translation.ha"],
   "aligner": {"model": "Davlan/afro-xlmr-large"}},
  {"path": "pairs/en-am.jsonl", "columns": ["en", "am"]}
]
```

## How it works

1. **BPE.** A byte-level BPE is trained on both sides of every pair. Reserved tokens come first.
2. **Alignment.** Each pair goes through the aligner (its source's own, if it has one). A BPE subword gets the mean
   vector of the aligner's word pieces that overlap it in characters. Two subwords are linked when each one's softmax
   over the other side's similarities exceeds `threshold` (awesome-align's rule).
3. **Scores.** The Dice score of two subwords is `2 * links / (occurrences(a) + occurrences(b))`. Identical subwords on
   the two sides (names, numbers) are not linked.
4. **Grouping.** Links are visited by descending Dice, skipping pairs below `min_link_count` or `min_dice`:
   - if neither subword is grouped, they start a new group;
   - if one is, the other joins its group unless the group already has `max_group_size` members;
   - two existing groups are never merged.

   *Every remaining token, including the reserved tokens, is a group of its own.*

## Files

`save_pretrained` writes:
- `tokenizer.json`: a standard `tokenizers` file;
- `tokenizer_config.json`: for `transformers.AutoTokenizer`;
- `groups.json`:

```json
{"max_group_size": 8, "token_group": [0, 1, 2, 3, ...], "token_member": [0, 0, 0, 0, ...],
 "groups": {"412": ["Ġhouse", "Ġmaison", "ĠHaus"], ...}}
```

`groups` is only there for people reading the file. It lists the groups with several members as raw BPE tokens, where
`Ġ` marks a leading space. 
`Grouping.load` reads `token_group` and `token_member`, and ignores any other keys.

## Development

Clone the repo
```bash
git clone https://github.com/henryhale/grouptok.git
cd grouptok
```

Install dependencies
```bash
pip install torch --index-url https://download.pytorch.org/whl/cpu   # optional: CPU torch is enough for the tests
pip install -e ".[train,test]"
pytest
```

The tests run offline. They train on a 500-pair English–French sample in `tests/data` (kept in the repository, not in
the package) with a stand-in aligner instead of a downloaded model.

Preview the [docs site](https://henryhale.github.io/grouptok/), built from this README and the docstrings
```bash
pip install -e ".[docs]"
mkdocs serve
```

## License

Released under the Apache-2.0 license. See [LICENSE](https://github.com/henryhale/grouptok/blob/master/LICENSE) for details.

&copy; 2026-present [Henry Hale](https://github.com/henryhale)

<!-- --8<-- [end:docs] -->
