Metadata-Version: 2.4
Name: ubpe-native
Version: 0.1.1
Summary: Python implementayion of UBPE tokenization algorithm.
License: MIT
License-File: LICENSE
Author: Scurrra (Ilja Baroŭski)
Author-email: iscurrra@gmail.com
Requires-Python: >=3.12
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: Implementation
Project-URL: Issues, https://github.com/Scurrra/ubpe-native/issues
Project-URL: Repository, https://github.com/Scurrra/ubpe-native.git
Description-Content-Type: text/markdown

# ubpe-native

Plain Python implementation of the Universal Byte Pair Encoding Tokenizer.

> The package is a part of the general [`ubpe`](https://github.com/Scurrra/ubpe) package, where I divided general import and implementations, because I'm planning to provide other implementations as well. So the package should not be directly installed. Please, use `pip install ubpe[native]` instead.
