Metadata-Version: 2.4
Name: simple_tokenizer
Version: 0.1.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: 3
License-File: LICENSE
Summary: Fast text tokenizer built with Rust and PyO3
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# Simple Tokenizer

Fast text tokenizer built with Rust and PyO3.

## Features

- `word_tokenizer`: Extract words from text using regex
- `sentence_tokenizer`: Split text into sentences

## Installation

```bash
pip install simple_tokenizer
```

## Usage

```python
import simple_tokenizer

text = "Hello world! This is a test."
tokens, count, elapsed = simple_tokenizer.word_tokenizer(text)
print(tokens)
```

