Metadata-Version: 2.1
Name: TokenizerChanger
Version: 1.0.3
Summary: Library for manipulating the existing tokenizer.
Home-page: https://github.com/1kkiRen/Tokenizer-Changer
Author: 1kkiren
Author-email: 1kkiren@mail.ru
Project-URL: GitHub, https://github.com/1kkiRen/Tokenizer-Changer
Keywords: tokenizer deletion tokens
Classifier: Programming Language :: Python :: 3.10
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tokenizers >=0.20.1
Requires-Dist: tqdm >=4.66.4
Requires-Dist: transformers >=4.45.2

# Tokenizer-Changer

Python script for manipulating the existing tokenizer.

The solution was tested on Llama3-8B tokenizer.

# Installation

Installation from PyPI:

```bash
pip install tokenizerchanger
```

# Requirements

- Python 3.9+
- tokenizers>=0.21.0
- transformers>=4.47.0
- tqdm>=4.66.4

# Docs

[https://tokenizer-changer.readthedocs.io/en/latest/](https://tokenizer-changer.readthedocs.io/en/latest/)
