Metadata-Version: 2.4
Name: basetenkenizer
Version: 0.2.6
Requires-Dist: numpy>=1.23.0
Requires-Dist: tiktoken>=0.13.0 ; extra == 'tiktoken'
Provides-Extra: tiktoken
License-File: LICENSE
License-File: NOTICES.txt
Summary: High-performance tokenization for inference
Home-Page: https://www.baseten.co
Author: Baseten
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://www.baseten.co
Project-URL: Repository, https://github.com/baseten/basetenkenizer

# Baseten Tokenizer

High-performance tokenization for inference, backed by Rust.

## Install

```bash
pip install basetenkenizer
```

## Standalone

```python
from basetenkenizer import Tokenizer

tokenizer = Tokenizer.from_model("deepseek-ai/DeepSeek-V3.2")
encoding = tokenizer.encode(
    "A very long prompt that is now much faster.",
    add_special_tokens=False,
)
print(encoding.ids)
```

## Kimi K3 rendered prompts

For a rendered prompt, `encode_segments` keeps template control text separate
from user content. Each segment is `(text, allow_special)`: use `True` for
template-generated text where added/control tokens should be recognized, and
`False` for user or tool content where those strings must remain ordinary text.
The default `tiktoken_safe=True` preserves the chunking behavior used by the
Kimi tokenizer.

```python
segments = [
    ("<|im_start|>user\n", True),
    (user_message, False),
    ("<|im_end|>\n<|im_start|>assistant\n", True),
]
encoding = tokenizer.encode_segments(segments, add_special_tokens=False)
token_ids = encoding.ids
```

## Transformers

```python
import basetenkenizer

basetenkenizer.patch_transformers()

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-120b")
tokens = tokenizer("Hello, world!")
```

Baseten Tokenizer is focused on inference and does not implement every feature
of Hugging Face Tokenizers.

Learn more about fast inference at [baseten.co](https://www.baseten.co).

## License

MIT

