Metadata-Version: 2.4
Name: rishe
Version: 0.1.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: Persian
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
License-File: LICENSE-APACHE
License-File: LICENSE-MIT
License-File: NOTICE.md
Summary: Rishe (ریشه): Blazingly fast, unified Persian NLP engine written in Rust — tokenization, SBD, lemmatization, ezafe detection, informal→formal normalization
Keywords: nlp,persian,farsi,tokenization,lemmatization,morphology
Home-Page: https://github.com/MahdiAlikhah/rishe
Author-email: Mahdi Alikhah <96587315+MahdiAlikhah@users.noreply.github.com>
License: MIT OR Apache-2.0
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Bug Tracker, https://github.com/MahdiAlikhah/rishe/issues
Project-URL: Documentation, https://docs.rs/rishe
Project-URL: Homepage, https://github.com/MahdiAlikhah/rishe
Project-URL: Repository, https://github.com/MahdiAlikhah/rishe

# Rishe (ریشه): fast Persian text processing for Python, built in Rust

`pip install rishe` gives you Persian normalization, tokenization, sentence
splitting, lemmatization, ezafe detection and informal → formal conversion
from one Rust core: no models to download, prebuilt wheels for Linux, macOS
and Windows. The core is also available as a Rust crate.

- Normalization is ~70× faster than Hazm and Shekar; tokenization is on par
  with them. See [docs/BENCHMARKS.md](docs/BENCHMARKS.md) for the full
  numbers, including where Rishe loses.
- `Pipeline.pipe()` processes batches in parallel (rayon) with the GIL
  released.

**Status: alpha (0.1).** Rishe is a fast preprocessing layer, not a
replacement for Hazm or Shekar: there is no POS tagging, NER, parsing,
spell checking or embeddings.

## Install

```bash
pip install rishe        # Python >= 3.9
```

or, from Rust (`Cargo.toml`): `rishe = "0.1"`

## Quick start

```python
from rishe import Pipeline

pipe = Pipeline()
doc = pipe("اون داره تو خونه کتاب میخونه، چیه؟")
doc.tokens       # ['آن', 'دارد', 'تو', 'خانه', 'کتاب', 'می‌خواند', '،', 'چیست', '؟']
doc.lemmas       # ['آن', 'داشت', 'تو', 'خانه', 'کتاب', 'خواند', '،', 'چیست', '؟']
doc.formal_text  # 'آن دارد تو خانه کتاب می‌خواند، چیست؟'
doc.sentences, doc.normalized_text, doc.ezafe_markers

pipe.word_tokenize("سلام، دنیا!")   # ['سلام', '،', 'دنیا', '!']
pipe.lemmatize("می‌روم")             # 'رفت'  (past-stem lemmas, UD style)
pipe.pipe(texts)                     # parallel batch -> list[ProcessedDoc]
```

```rust
use rishe::Pipeline;

let pipe = Pipeline::create(true, true, true); // unify digits, strip diacritics, fix ZWNJ
let doc = pipe.process("کتاب‌های خوب را به دوستانم می‌دهم.");
println!("{:?}", doc.tokens);
```

## Known limitations

- `Pipeline.__call__` also rewrites colloquial words to formal ones, so
  `doc.tokens` can differ from the surface text. Use `word_tokenize` for
  plain tokenization.
- Informal → formal conversion is lexicon- and rule-based. Common forms
  (`اون`, `خونه`, `داره`, `چیه`, `میخوام`) work; many suffixed or rarer forms
  (`خونه‌ام`, `کتابه`) are left unchanged.
- Lemmas follow the Universal Dependencies convention (verbs lemmatize to the
  past stem). Accuracy on UD is ~0.90 (Seraji) and ~0.88 (PerDT).
- Ezafe detection is rule-based and has not been evaluated.

## Building from source

```bash
cargo test
maturin develop --release     # Python extension into the current venv
python scripts/benchmark.py   # needs: hazm shekar conllu
```

`scripts/extract_lexicons.py` regenerates `data/*.tsv` from Hazm's data
files; `build.rs` compiles them into FSTs.

## License

Dual-licensed under [Apache-2.0](LICENSE-APACHE) or [MIT](LICENSE-MIT), at
your option. The bundled lexicons are derived from
[Hazm](https://github.com/roshan-research/hazm) data (MIT); see
[NOTICE.md](NOTICE.md).

