Metadata-Version: 2.4
Name: rag-tokenizer-for-ragflow
Version: 0.0.0
Summary: rag tokenizer for RAGFlow
Requires-Python: <3.15,>=3.10
Description-Content-Type: text/markdown
Requires-Dist: readerwriterlock<2.0.0,>=1.0.0
Requires-Dist: datrie<0.9.0,>=0.8.3
Requires-Dist: hanziconv<0.4.0,>=0.3.0
Requires-Dist: nltk<4.0.0,>=3.9.0

# rag-tokenizer

# Upload to pypi.org 

```shell
uv build;
uv publish --token $PYPI_API_TOKEN
```

# Install
```shell
pip install rag-tokenizer
```

# Usage

```python
import rag_tokenizer
res = rag_tokenizer.tokenize("Hello world!")
print(res)
```
