Metadata-Version: 2.4
Name: mothertoken
Version: 0.1.0.0a1
Summary: Benchmarking tool for tokenizer efficiency across languages and models
Project-URL: Homepage, https://github.com/inimaz/mothertoken
Project-URL: Repository, https://github.com/inimaz/mothertoken
Project-URL: Issues, https://github.com/inimaz/mothertoken/issues
Author: Inigo
License-Expression: MIT
License-File: LICENSE
Keywords: benchmark,fairness,llm,multilingual,tokenizer
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.11
Requires-Dist: anthropic>=0.25.0
Requires-Dist: datasets>=2.14.0
Requires-Dist: google-generativeai>=0.5.0
Requires-Dist: huggingface-hub>=0.20.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: rich>=13.0.0
Requires-Dist: tiktoken>=0.7.0
Requires-Dist: transformers>=4.40.0
Requires-Dist: typer>=0.12.0
Description-Content-Type: text/markdown

# mothertoken

> *Every model has a native tongue. The question is whether yours matches.*

Benchmarking tool and dataset exploring how tokenizer design creates silent efficiency, quality, and carbon inequities for non-English languages.

> [!NOTE]
> **Alpha release:** tokenizer coverage is still expanding. The current tokenizer list is useful for comparison, but it is not complete yet.

## Installation

```bash
# Published package
pip install mothertoken
```

For local development:

```bash
git clone https://github.com/inimaz/mothertoken
cd mothertoken
uv sync
uv pip install -e .
```

## CLI Usage

The `mothertoken` command is available after installation.

### 📊 Rank tokenizers for a language
Rank supported tokenizers for a specific language using the precomputed benchmark data.
```bash
mothertoken rank spanish

# Raw FLORES+ codes still work
mothertoken rank spa_Latn
```

### 🧭 List tokenizers
See which tokenizer IDs can be used and which familiar models use them.
```bash
mothertoken list

# Only local tokenizers
mothertoken list --local-only
```

### 🔤 Tokenize exact text
Count tokens for exact text using local tokenizers by default. Add `--language` to estimate the English-equivalent token count from the benchmark multiplier.
```bash
mothertoken tokenize "Hola Mundo" --language es

# Check one model
mothertoken tokenize "Hello" --model gpt-4o

# Include API-backed provider token counters when API keys are configured
mothertoken tokenize "Hello" --include-api
mothertoken tokenize "Hello" --model claude-sonnet --include-api

# Estimate the English-equivalent count for a known language
mothertoken tokenize "مرحبا بالعالم" --language ar --model gpt-4o

# Compare against your own English translation
mothertoken tokenize "مرحبا بالعالم" --language ar --english-text "Hello world"

# Tokenize a file
mothertoken tokenize --file prompt.txt

# Compare translated files
mothertoken tokenize --file prompt.ar.txt --language ar --english-file prompt.en.txt
```

---

## Researcher Workflow

Benchmark regeneration and model-extension docs live in [`docs/benchmarking.md`](docs/benchmarking.md).

## License
MIT
