Metadata-Version: 2.4
Name: mbpe
Version: 0.1.1
Summary: BPE tokenizer — Mojo implementation with Python bindings
Author: tenmoomnet
License: MIT
Project-URL: Homepage, https://github.com/tenmoomnet/mbpe
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# mbpe

BPE tokenizer — Mojo implementation with Python bindings.

```python
import mbpe
tok = mbpe.get_encoding("gpt2")
tok.encode("hello world")
```

Supports gpt2, cl100k, o200k encodings plus custom training via GPreTokenizer, GPT2Tokenizer, GPT4Tokenizer.
