Metadata-Version: 2.4
Name: gigatoken
Version: 0.3.0
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Requires-Dist: awkward>=2.6.3
Requires-Dist: numpy>=1.24.4
License-File: LICENSE
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# Gigatoken

<div align="center">

400x faster than HuggingFace, drop-in replacement.

*Tokenize your text data at GB/s!*
</div>


## Install
```
pip install gigatoken
```


## Usage
```
import gigatoken as gt

hf_tokenizer = ...



```

## How does Gigatoken work?

Gigatoken came from a few observations:
* A majority of the time in current tokenizers is spent on pretokenization -> 

Gigatoken implements pretokenizers that run at >2GB/s/thread

Gigatoken is faster than other libraries due to algorithmic and systems changes.
One key benefit of using Gigatoken is that it replaces the regex expression used by almost all tokenizers to do pre-tokenization with a custom implementation of the exact same method.
This is a serious bottleneck for other implementations, and is a big part in this library's breakneck speeds.
Additionally, Gigatoken uses concurrent data structures to use multiprocessing in more places.

\* All reference speeds in this section are measured on an M4 Pro CPU
