Metadata-Version: 2.4
Name: tokenizers
Version: 1.0.0rc4
Requires-Dist: huggingface-hub>=0.16.4,<3.0
Requires-Dist: numpy
Summary: Reworked tokenizers Python bindings, over the pipeline encode path (not for release yet)
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# tokenizers Python bindings

This module is still experimental, expect the API to change or break.
If used in a production setting, please pin the version number.

## Install

```
pip install --pre tokenizers
```


## Usage

```python
from tokenizers import Padding, Tokenizer

# A `Tokenizer` represents a tokenizer's spec (json) and can be edited.
# You can load it from the JSON file
tokenizer = Tokenizer.from_file("tokenizer.json")
# Or load the `tokenizer.json` of a model repo on the Hub.
# Downloads go through the `huggingface_hub` package
tokenizer = Tokenizer.from_pretrained("openai-community/gpt2", revision="main")

# `compile()` compiles the specification to an optimized tokenizer in rust.
# The resulting `CompiledTokenizer` object can encode text to token ids and decode ids back to text.
# It's immutable and can be shared among threads.
compiled = tokenizer.compile()

encoding = compiled.encode("Hello there, how are you?")
encoding.ids             # [101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102]
encoding.type_ids        # [0, 0, 0, 0, 0, 0, 0, 0, 0]
encoding.attention_mask  # [1, 1, 1, 1, 1, 1, 1, 1, 1]

# The same fields as numpy arrays. Each read returns a *new copy* that the caller owns.
encoding.ids_array             # array([101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102], dtype=uint32)
encoding.type_ids_array        # array([0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint32)
encoding.attention_mask_array  # array([1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=uint32)
torch.from_numpy(encoding.ids_array)  # no-copy translation

compiled.encode("Hello", add_special_tokens=False).ids  # no [CLS]/[SEP]
compiled.decode(encoding.ids)                           # "hello there, how are you?"
compiled.decode(encoding.ids_array)                     # a numpy array works too
compiled.decode(encoding.ids, skip_special_tokens=False)

# Pad one call. `length=None` pads each batch to its longest item.
short, long = compiled.encode_batch(
    ["Hello", "Hello there, how are you?"],
    padding=Padding(direction="left", pad_id=0, pad_token="[PAD]"),
)
len(short) == len(long)  # True
short.attention_mask     # [0, 0, 0, 0, 0, 0, 1, 1, 1]
compiled.encode("Hello", padding=Padding(length=16))  # exactly 16 ids

# Padding and truncation can also be set on the spec.
# `padding=None` switches the default off for one call.
tokenizer.padding = Padding(pad_to_multiple_of=8)
padded = tokenizer.compile()
padded.encode("Hello", padding=None)
```

### Edit and save a tokenizer

```python
from tokenizers import AddedToken, normalizers

# Set or replace a component. Reading `tokenizer.normalizer` returns a live view, not a copy.
tokenizer.normalizer = normalizers.Sequence([normalizers.NFC(), normalizers.Lowercase()])

# Added tokens are matched in the text before the model runs.
tokenizer.add_special_tokens(["<|im_end|>"])
tokenizer.add_tokens([AddedToken("<think>", single_word=True)])

tokenizer.save("tokenizer.json")
compiled = tokenizer.compile()  # compile again to encode with the edits
compiled.token_to_id("<think>")  # a new token gets the next free id
compiled.get_vocab_size()        # counts the added tokens too
```

`examples/composition.py` builds a tokenizer from a model, then edits the model after the tokenizer holds it.

### Decode a stream of ids

`DecodeStream` turns ids into text as they arrive, such as a model's generated tokens.
`step` returns `None` while the text ends with an incomplete character.

```python
from tokenizers.decoders import DecodeStream

stream = DecodeStream(skip_special_tokens=True)
for id in generated_ids:
    print(stream.step(compiled, id) or "", end="")
```

`examples/` holds a runnable script for each of these; `make examples` runs them all.

## multiprocessing and Worker processes

Every class here can be pickled, so a tokenizer can be sent to a worker process and its encodings
sent back: `multiprocessing`, a PyTorch `DataLoader`, `datasets.map(num_proc=...)`.

A pickled `CompiledTokenizer` carries its whole `tokenizer.json`, and unpickling compiles it again.
With the `"spawn"` start method, we recommend compiling the tokenizer once per worker to avoid unpickle compilation overhead.

```python
def compile_tokenizer(tokenizer):
    global compiled
    compiled = tokenizer.compile()


def encode(text):
    return compiled.encode(text)


with multiprocessing.get_context("spawn").Pool(4, initializer=compile_tokenizer, initargs=(tokenizer,)) as pool:
    encoded = list(pool.imap(encode, texts, chunksize=32))
```

With the `"fork"` start method, compiling before starting the pool is recommended: the workers inherit the compiled tokenizer
and nothing is pickled.

## Developers

- `make develop` regenerates the `.pyi` stub, rebuilds the extension (debug) and reinstalls it
  in editable mode.
- `make test` does the above, then runs `tests/` with pytest.
- `make examples` does the above, then runs every script in `examples/`.
- `make stubs` regenerates the `.pyi` stubs in `python/tokenizers/` (`tokenizers.pyi`,
  `models.pyi`, `normalizers.pyi`, ...), including the members `python/tokenizers/__init__.py`
  attaches to the Rust classes.
- `make style` / `make check-style` regenerate the stub, then format/lint the Rust and
  Python sides and type-check with `ty`. CI runs `make check-style` and fails if the
  regenerated stub differs from the committed one.

