Metadata-Version: 2.4
Name: gigatok
Version: 0.2.0
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Requires-Dist: awkward>=2.6.3
Requires-Dist: numpy>=1.24.4
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# Gigatok


<div align="center">

400x faster than HuggingFace, drop-in replacement.

*Tokenize your text data at GB/s!*
</div>


# Install
```
pip install gigatok
```


# How does Gigatok work?

Gigatok came from a few observations:
* A majority of the time in current tokenizers is spent on pretokenization -> 

Gigatok implements pretokenizers that run at >2GB/s/thread

Gigatok is faster than other libraries due to algorithmic and systems changes.
One key benefit of using Gigatok is that it replaces the regex expression used by almost all tokenizers to do pre-tokenization with a custom implementation of the exact same method.
This is a serious bottleneck for other implementations, and is a big part in this library's breakneck speeds.
Additionally, Gigatok uses concurrent data structures to use multiprocessing in more places.

\* All reference speeds in this section are measured on an M4 Pro CPU
