Metadata-Version: 2.4
Name: tokenizers
Version: 1.0.0rc5
Requires-Dist: huggingface-hub>=0.16.4,<3.0
Summary: Reworked tokenizers Python bindings, over the pipeline encode path (not for release yet)
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# tokenizers Python bindings

This module is still experimental, expect the API to change or break.
If used in a production setting, please pin the version number.

## Install

```
pip install --pre tokenizers
```


## Usage

```python
from tokenizers import Padding, Tokenizer

# A `Tokenizer` represents a tokenizer's spec (json) and can be edited.
# You can load it from the JSON file
tokenizer = Tokenizer.from_file("tokenizer.json")
# Or load the `tokenizer.json` of a model repo on the Hub.
# Downloads go through the `huggingface_hub` package
tokenizer = Tokenizer.from_pretrained("openai-community/gpt2", revision="main")

# `compile()` compiles the specification to an optimized tokenizer in rust.
# The resulting `CompiledTokenizer` object can encode text to token ids and decode ids back to text.
# It's immutable and can be shared among threads.
compiled = tokenizer.compile()

encoding = compiled.encode("Hello there, how are you?")
encoding.ids             # [101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102]
encoding.type_ids        # [0, 0, 0, 0, 0, 0, 0, 0, 0]
encoding.attention_mask  # [1, 1, 1, 1, 1, 1, 1, 1, 1]

# The same fields for array libraries, through DLPack. Each read is an `int64` copy.
torch.tensor(encoding.ids_dlpack)   # tensor([101, 7592, ...]), torch 2.6 or later
np.from_dlpack(encoding.attention_mask_dlpack)  # array([1, 1, ...])

compiled.encode("Hello", add_special_tokens=False).ids  # no [CLS]/[SEP]
compiled.decode(encoding.ids)                           # "hello there, how are you?"
compiled.decode(encoding.ids, skip_special_tokens=False)

# Pad one call. `length=None` pads each batch to its longest item.
short, long = compiled.encode_batch(
    ["Hello", "Hello there, how are you?"],
    padding=Padding(direction="left", pad_id=0, pad_token="[PAD]"),
)
len(short) == len(long)  # True
short.attention_mask     # [0, 0, 0, 0, 0, 0, 1, 1, 1]
compiled.encode("Hello", padding=Padding(length=16))  # exactly 16 ids

# Padding and truncation can also be set on the spec.
# `padding=None` switches the default off for one call.
tokenizer.padding = Padding(pad_to_multiple_of=8)
padded = tokenizer.compile()
padded.encode("Hello", padding=None)
```

`examples/` holds a runnable script for each of these; `make examples` runs them all.

## multiprocessing and Worker processes

Every class here can be pickled, so a tokenizer can be sent to a worker process and its encodings
sent back: `multiprocessing`, a PyTorch `DataLoader`, `datasets.map(num_proc=...)`.

A pickled `CompiledTokenizer` carries its whole `tokenizer.json`, and unpickling compiles it again.
With the `"spawn"` start method, we recommend compiling the tokenizer once per worker to avoid unpickle compilation overhead.

```python
def compile_tokenizer(tokenizer):
    global compiled
    compiled = tokenizer.compile()


def encode(text):
    return compiled.encode(text)


with multiprocessing.get_context("spawn").Pool(4, initializer=compile_tokenizer, initargs=(tokenizer,)) as pool:
    encoded = list(pool.imap(encode, texts, chunksize=32))
```

With the `"fork"` start method, compiling before starting the pool is recommended: the workers inherit the compiled tokenizer
and nothing is pickled.

## Developers

- `make develop` regenerates the `.pyi` stub, rebuilds the extension (debug) and reinstalls it
  in editable mode.
- `make test` does the above, then runs `tests/` with pytest.
- `make examples` does the above, then runs every script in `examples/`.
- `make stubs` regenerates `python/tokenizers/tokenizers.pyi`, including the members
  `python/tokenizers/__init__.py` attaches to the Rust classes.
- `make style` / `make check-style` regenerate the stub, then format/lint the Rust and
  Python sides and type-check with `ty`. CI runs `make check-style` and fails if the
  regenerated stub differs from the committed one.

