Metadata-Version: 2.4
Name: puretiktoken
Version: 0.1.0
Summary: tiktoken in pure Python: byte-for-byte compatible BPE tokenizer with no Rust extension
Author: adam2go
License: MIT
Project-URL: Homepage, https://github.com/adam2go/puretiktoken
Keywords: tiktoken,tokenizer,bpe,openai,gpt,llm,pure-python,token-counting,cl100k,o200k
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# puretiktoken

[![CI](https://github.com/adam2go/puretiktoken/actions/workflows/ci.yml/badge.svg)](https://github.com/adam2go/puretiktoken/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/puretiktoken)](https://pypi.org/project/puretiktoken/)
[![Python](https://img.shields.io/badge/python-3.9%E2%80%933.14%20%7C%20PyPy-blue)](.github/workflows/ci.yml)
[![tiktoken conformance](https://img.shields.io/badge/vs%20real%20tiktoken-byte--for--byte-brightgreen)](tests/test_conformance.py)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**[tiktoken](https://github.com/openai/tiktoken) in pure Python: byte-for-byte
compatible BPE tokenizer for OpenAI models — no Rust extension, no binary
wheel, zero dependencies.**

OpenAI's `tiktoken` is a Rust extension. That's fast, but it means a compiled
wheel has to exist (and load) for your platform — which it doesn't on
Pyodide/WASM, often fails on fresh ARM or musl/Alpine boxes, bloats AWS Lambda
layers, and won't build in locked-down sandboxes with no Rust toolchain.
puretiktoken is the same tokenizer written in plain Python, so it **runs
anywhere Python runs** and produces the **exact same token ids**.

```sh
pip install puretiktoken
```

```python
import puretiktoken as tiktoken      # drop-in: same names, same ids

enc = tiktoken.get_encoding("cl100k_base")
enc.encode("hello world")             # -> [15339, 1917]   (identical to tiktoken)
enc.decode([15339, 1917])             # -> "hello world"

tiktoken.encoding_for_model("gpt-4o").encode("tokens!")   # o200k_base
len(tiktoken.get_encoding("o200k_base").encode(prompt))    # count before an API call
```

The API mirrors the slice of `tiktoken` people actually use —
`get_encoding`, `encoding_for_model`, `encode`, `encode_ordinary`, `decode`,
`decode_bytes`, `n_vocab`, plus `allowed_special` / `disallowed_special`
handling — so for token counting and offline tokenization it is a drop-in
replacement.

## Why pure Python

`tiktoken` (Rust), Hugging Face `tokenizers` (Rust) and `sentencepiece` (C++)
are all native extensions — there was **no pure-Python tokenizer that matches
tiktoken's output**. The hard part isn't the BPE merge (that's ~30 lines); it's
tiktoken's pre-tokenizer, a `fancy-regex` pattern using `\p{L}`/`\p{N}` Unicode
classes, **possessive quantifiers** and a **lookahead** — features the stdlib
`re` can't even compile. puretiktoken reproduces that splitter exactly with a
hand-written scanner over `unicodedata` categories, then runs the same
rank-based merge. The result is verified byte-for-byte against the real
`tiktoken` (see below).

Zero dependencies, zero binaries: the vocabularies ship gzipped inside the
wheel, so it works **fully offline** — no download of `*.tiktoken` files at
runtime, which is the other thing that breaks tiktoken in sandboxed or
air-gapped environments.

## Speed, stated honestly

Pure Python can't beat a Rust extension, and this doesn't try to. It encodes at
roughly **0.7–1.3 M tokens/sec** (CPython 3.12, single core), which is **~7×
slower than `tiktoken` on `cl100k_base` and ~15× on `o200k_base`** — and still
fast enough that token counting is never your bottleneck.

| | puretiktoken (CPython 3.12) | tiktoken (Rust) |
|---|---:|---:|
| `cl100k_base` | ~1.25 M tok/s | ~5.7 M tok/s |
| `o200k_base` | ~0.71 M tok/s | ~8.1 M tok/s |

The point is not speed — it's **getting tiktoken's exact ids where tiktoken
won't install or run.** If you need both, use `tiktoken` when its wheel loads
and fall back to `puretiktoken` when it doesn't; the ids are identical.

## Verified against the real tiktoken

Conformance is differential, the same way [purere2](https://github.com/adam2go/purere2)
checks itself against the real RE2 and [purefzf](https://github.com/adam2go/purefzf)
against the `fzf` binary: text is tokenized by both puretiktoken and `tiktoken`
and the token ids are compared. Across a curated multilingual/code/emoji corpus
plus **tens of thousands of randomized inputs over every assigned Unicode
codepoint**, the output is **byte-for-byte identical** for both `cl100k_base`
and `o200k_base` — including the tricky parts: contractions with Unicode case
folding (`'ſ` folds to `'s`), `o200k`'s camelCase splitting (`iPhone` →
`i`+`Phone`), combining marks, and whitespace-run edges. Decoding round-trips
losslessly for arbitrary input. The test suite locks this, so any regression
fails CI.

The one place puretiktoken and tiktoken *can* differ is **unassigned**
codepoints. tiktoken's Rust regex bundles its own Unicode database; puretiktoken
uses your Python's `unicodedata`. When those versions disagree about whether a
brand-new codepoint is a letter, the split can differ — but only for text that
contains codepoints unassigned in your Python, which real-world text never does.
The conformance suite asserts this explicitly: every divergence must be
explained by an unassigned codepoint; a mismatch on assigned text is treated as
a bug.

## Encodings

`cl100k_base` (GPT-4, GPT-3.5, `text-embedding-3-*`, ada-002) and `o200k_base`
(GPT-4o, GPT-4o-mini, o1, o3). `encoding_for_model()` maps model names to these,
matching tiktoken's mapping for the current model families.

## License

[MIT](LICENSE). The bundled vocabularies are OpenAI's, distributed under the
MIT terms of [tiktoken](https://github.com/openai/tiktoken).
