Metadata-Version: 2.4
Name: mothertoken
Version: 0.1.0.0a2
Summary: Benchmarking tool for tokenizer efficiency across languages and models
Project-URL: Homepage, https://github.com/inimaz/mothertoken
Project-URL: Repository, https://github.com/inimaz/mothertoken
Project-URL: Issues, https://github.com/inimaz/mothertoken/issues
Author: Inigo
License-Expression: MIT
License-File: LICENSE
Keywords: benchmark,fairness,llm,multilingual,tokenizer
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.11
Requires-Dist: anthropic>=0.25.0
Requires-Dist: datasets>=2.14.0
Requires-Dist: google-generativeai>=0.5.0
Requires-Dist: huggingface-hub>=0.20.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: rich>=13.0.0
Requires-Dist: tiktoken>=0.12.0
Requires-Dist: transformers>=5.7.0
Requires-Dist: typer>=0.12.0
Description-Content-Type: text/markdown

<p align="center">
  <img src="web/public/favicon.svg" alt="mothertoken logo" width="96" height="96">
</p>

# mothertoken

> *Every model has a native tongue. The question is whether yours matches.*

Toolkit for comparing tokenizer efficiency across languages, model families, and user-supplied Hugging Face refs.

> [!NOTE]
> **Alpha release:** the bundled benchmark is curated and representative, not exhaustive. Use direct Hugging Face refs when you want to compare tokenizers outside the starter set.

## Installation

```bash
# Published package
pip install mothertoken
```

For local development:

```bash
git clone https://github.com/inimaz/mothertoken
cd mothertoken
uv sync
uv pip install -e .
```

## CLI Usage

The `mothertoken` command is available after installation.

### Common questions

| Question | Command |
| --- | --- |
| I speak a language that is not English. Which tokenizer is most efficient for it? | `mothertoken rank spanish` |
| I have this text and a chosen model. How many tokens does it use? | `mothertoken tokenize "Hola Mundo" --model gpt-4o` |
| I am choosing between a few models. Which one tokenizes my text best? | `mothertoken compare "Travesura realizada" --model gpt-oss --model Qwen/Qwen3-0.6B` |
| How good are the known models at tokenizing my language? | `mothertoken rank arabic` |
| I have this model. Which languages does it tokenize best, which ones worst? | `mothertoken benchmark --models gpt-oss,YOUR_MODEL1,YOUR_MODEL2` |

### Rank tokenizers for a language
Rank supported tokenizers for a specific language using the precomputed benchmark data.
```bash
mothertoken rank spanish

# Raw FLORES+ codes still work
mothertoken rank spa_Latn
```

### List tokenizers
See which tokenizer IDs can be used and which familiar models use them.
```bash
mothertoken list
```

### Tokenize exact text
Count tokens for exact text using local tokenizers by default. Add `--language` to estimate the English-equivalent token count from the benchmark multiplier.
```bash
mothertoken tokenize "Hola Mundo" --language es

# Check one model
mothertoken tokenize "Hello" --model gpt-4o

# Check a Hugging Face model/tokenizer ref directly
mothertoken tokenize "Hello" --model Qwen/Qwen3-0.6B

# Estimate the English-equivalent count for a known language
mothertoken tokenize "مرحبا بالعالم" --language ar --model gpt-4o

# Compare against your own English translation
mothertoken tokenize "مرحبا بالعالم" --language ar --english-text "Hello world"

# Tokenize a file
mothertoken tokenize --file prompt.txt

# Compare translated files
mothertoken tokenize --file prompt.ar.txt --language ar --english-file prompt.en.txt
```

### Compare selected tokenizers
Compare aliases from `mothertoken list` with direct Hugging Face refs. This is the main workflow when you care about a specific set of models.
```bash
mothertoken compare "Travesura realizada" \
  --model gpt-4o \
  --model Qwen/Qwen3-0.6B \
  --model mistralai/Mistral-7B-v0.1

mothertoken compare --file prompt.txt \
  --model mistralai/Mistral-7B-v0.1 \
  --model deepseek-ai/DeepSeek-V4-Pro
```

---

## Researcher Workflow

Benchmark regeneration and model-extension docs live in [`docs/benchmarking.md`](docs/benchmarking.md).

You can also benchmark a direct Hugging Face ref without adding it to `default_tokenizers.yaml`:

```bash
uv run mothertoken benchmark --languages eng_Latn,arb_Arab --models Qwen/Qwen3-0.6B
```

## License
MIT
