Metadata-Version: 2.1
Name: pyregtokenizer
Version: 0.0.2
Summary: A BPE Tokenizer using regex
Author: Pratyaksh Agarwal
Author-email: pratyakshagarwal93@gmail.com
Keywords: python,bpe,tokenizer,regex
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: Unix
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Description-Content-Type: text/plain
Requires-Dist: regex

A BPE Tokenizer using regex
