Metadata-Version: 2.4
Name: puretiktoken
Version: 0.2.1
Summary: tiktoken in pure Python: byte-for-byte compatible BPE tokenizer with no Rust extension
Author: adam2go
License: MIT
Project-URL: Homepage, https://github.com/adam2go/puretiktoken
Keywords: tiktoken,tokenizer,bpe,openai,gpt,llm,pure-python,token-counting,cl100k,o200k
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# puretiktoken

[![CI](https://github.com/adam2go/puretiktoken/actions/workflows/ci.yml/badge.svg)](https://github.com/adam2go/puretiktoken/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/puretiktoken)](https://pypi.org/project/puretiktoken/)
[![Python](https://img.shields.io/badge/python-3.9%E2%80%933.14%20%7C%20PyPy-blue)](.github/workflows/ci.yml)
[![tiktoken conformance](https://img.shields.io/badge/vs%20real%20tiktoken-byte--for--byte-brightgreen)](tests/test_conformance.py)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**[tiktoken](https://github.com/openai/tiktoken) in pure Python: byte-for-byte
compatible BPE tokenizer for OpenAI models — no Rust extension, no binary
wheel, zero dependencies.**

OpenAI's `tiktoken` is a Rust extension. That's fast, but it means a compiled
wheel has to exist (and load) for your platform — which it doesn't on
Pyodide/WASM, often fails on fresh ARM or musl/Alpine boxes, bloats AWS Lambda
layers, and won't build in locked-down sandboxes with no Rust toolchain.
puretiktoken is the same tokenizer written in plain Python, so it **runs
anywhere Python runs** and produces the **exact same token ids**.

```sh
pip install puretiktoken
```

```python
import puretiktoken as tiktoken      # drop-in: same names, same ids

enc = tiktoken.get_encoding("cl100k_base")
enc.encode("hello world")             # -> [15339, 1917]   (identical to tiktoken)
enc.decode([15339, 1917])             # -> "hello world"

tiktoken.encoding_for_model("gpt-4o").encode("tokens!")   # o200k_base
len(tiktoken.get_encoding("o200k_base").encode(prompt))    # count before an API call
```

The API mirrors the slice of `tiktoken` people actually use —
`get_encoding`, `encoding_for_model`, `encode`, `encode_ordinary`, `decode`,
`decode_bytes`, `n_vocab`, plus `allowed_special` / `disallowed_special`
handling — so for token counting and offline tokenization it is a drop-in
replacement.

## Why pure Python

`tiktoken` (Rust), Hugging Face `tokenizers` (Rust) and `sentencepiece` (C++)
are all native extensions — there was **no pure-Python tokenizer that matches
tiktoken's output**. The hard part isn't the BPE merge (that's ~30 lines); it's
tiktoken's pre-tokenizer, a `fancy-regex` pattern using `\p{L}`/`\p{N}` Unicode
classes, **possessive quantifiers** and a **lookahead** — features the stdlib
`re` can't even compile. puretiktoken reproduces that splitter exactly: it
translates each character to a single-byte Unicode *class code* (the only
Unicode-aware step, done with `str.translate` at C speed), then runs an
equivalent pattern — now free of `\p{...}` — in the C `re` engine, and feeds
the pieces to the same rank-based merge. The result is verified byte-for-byte
against the real `tiktoken` (see below).

Zero dependencies, zero binaries: the vocabularies ship gzipped inside the
wheel, so it works **fully offline** — no download of `*.tiktoken` files at
runtime, which is the other thing that breaks tiktoken in sandboxed or
air-gapped environments.

## Speed, stated honestly

A Rust extension should win — yet on **real-world text `cl100k_base` matches or
beats `tiktoken`** here. puretiktoken translates each character to a one-byte
Unicode class and lets the C `re` engine do the splitting, then caches the ids
of each pre-token (real text repeats tokens heavily, by Zipf's law) and encodes
in one fused pass. It runs at **~4.5–5.7 M tokens/sec** on CPython 3.12 — faster
than `tiktoken` on `cl100k_base`, because it stays in-process while `tiktoken`
pays Python↔Rust overhead on every call.

| corpus (CPython 3.12) | `cl100k_base` | `o200k_base` |
| --- | ---: | ---: |
| prose / code / markdown | **~0.7–0.9× (beats tiktoken)** | ~1.3–1.6× tiktoken |
| adversarial high-entropy¹ | ~6× tiktoken | ~7× tiktoken |

¹ Random unique strings where no token ever repeats, so the cache can't help and
it falls back to the pure-Python BPE merge. Real text never looks like this; run
`python tools/bench.py` to measure on your own. (On older CPython the constant
factors are higher — e.g. ~1.1×/1.9× on 3.9 — and PyPy JITs it faster still.)

The point was never to beat Rust — it's **getting tiktoken's exact ids where
tiktoken won't install or run.** That it's now also fast on real text is a bonus.

## Verified against the real tiktoken

Conformance is differential, the same way [purere2](https://github.com/adam2go/purere2)
checks itself against the real RE2 and [purefzf](https://github.com/adam2go/purefzf)
against the `fzf` binary: text is tokenized by both puretiktoken and `tiktoken`
and the token ids are compared. Across a curated multilingual/code/emoji corpus
plus **tens of thousands of randomized inputs over every assigned Unicode
codepoint**, the output is **byte-for-byte identical** for both `cl100k_base`
and `o200k_base` — including the tricky parts: contractions with Unicode case
folding (`'ſ` folds to `'s`), `o200k`'s camelCase splitting (`iPhone` →
`i`+`Phone`), combining marks, and whitespace-run edges. Decoding round-trips
losslessly for arbitrary input. The test suite locks this, so any regression
fails CI.

The one place puretiktoken and tiktoken *can* differ is **unassigned**
codepoints. tiktoken's Rust regex bundles its own Unicode database; puretiktoken
uses your Python's `unicodedata`. When those versions disagree about whether a
brand-new codepoint is a letter, the split can differ — but only for text that
contains codepoints unassigned in your Python, which real-world text never does.
The conformance suite asserts this explicitly: every divergence must be
explained by an unassigned codepoint; a mismatch on assigned text is treated as
a bug.

## Encodings

`cl100k_base` (GPT-4, GPT-3.5, `text-embedding-3-*`, ada-002) and `o200k_base`
(GPT-4o, GPT-4o-mini, o1, o3). `encoding_for_model()` maps model names to these,
matching tiktoken's mapping for the current model families.

## License

[MIT](LICENSE). The bundled vocabularies are OpenAI's, distributed under the
MIT terms of [tiktoken](https://github.com/openai/tiktoken).
