Metadata-Version: 2.4
Name: ubpe-native
Version: 0.2.0b1
Summary: Python implementation of UBPE tokenization algorithm.
License: MIT
License-File: LICENSE
Author: Scurrra (Ilja Baroŭski)
Author-email: iscurrra@gmail.com
Requires-Python: >=3.11
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: Implementation
Project-URL: Issues, https://github.com/Scurrra/ubpe-native/issues
Project-URL: Repository, https://github.com/Scurrra/ubpe-native.git
Description-Content-Type: text/markdown

# ubpe-native

> Native Python implementation of the [Universal Byte Pair Encoding Tokenizer](https://github.com/Scurrra/ubpe).

> The package is a part of the general [`ubpe`](https://github.com/Scurrra/ubpe) package, where I divided general import and implementations, because I'm planning to provide other implementations as well. So the package should not be directly installed. Please, use `pip install ubpe[native]` instead.

## Contribution

If you know how to improve the code, especially how to make it faster, fill free to contribute.

