Metadata-Version: 2.4
Name: farsflow
Version: 0.1.1
Summary: Deterministic, semantic-safe Persian text preprocessing for AI, NLP, and search pipelines.
Author-email: Mahdi Hosseini <mhossza@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/mhhoss/farsflow
Project-URL: Documentation, https://github.com/mhhoss/farsflow#readme
Project-URL: Source, https://github.com/mhhoss/farsflow
Project-URL: Issues, https://github.com/mhhoss/farsflow/issues
Keywords: farsi,persian,preprocessing,nlp,text-cleaning,normalization,ai-pipelines,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# farsflow

farsflow is a lightweight **Persian text preprocessing** library focused on deterministic, semantic-safe normalization for modern AI and search pipelines.

[![PyPI version](https://img.shields.io/pypi/v/farsflow.svg)](https://pypi.org/project/farsflow/)
[![Python Versions](https://img.shields.io/pypi/pyversions/farsflow.svg)](https://pypi.org/project/farsflow/)

## ❓ WHY farsflow??

Persian text often contains inconsistent spacing, Arabic/Persian character variants, invisible Unicode formatting characters, Arabic diacritics, and broken ZWNJ usage that negatively affect search, embeddings, NLP pipelines, and LLM applications.

farsflow provides a minimal and deterministic preprocessing layer designed to clean text without aggressive or semantic-destructive transformations.

---

## 🚀 Features

- Deterministic and semantic-safe normalization
- Safe ZWNJ (Joiner) correction
- Whitespace and punctuation cleanup
- Unicode cleanup (Arabic/Persian variants, Bidi controls, invisible formatting characters)
- Configurable normalization options (digit normalization & diacritics removal)
- Modular processors
- Zero dependencies

---

## 📦 Installation

```bash
pip install farsflow
```

## ✨ Quick Start

```python
import farsflow as ff

text = "سلام  دنیا!  این یك   تست است  که می نویسم  ۴۵۶"
cleaned = ff.clean(text)
print(cleaned)
```
Expected output:
```text
سلام دنیا! این یک تست است که می‌نویسم 456
```

---

## 🧩 Pipeline Components

farsflow ships with a set of modular, composable components:

- **Normalizer** — character normalization, Unicode cleanup, and optional diacritics removal
- **JoinerFixer** — fixes ZWNJ usage without over-correction
- **SpaceCleaner** — trims redundant whitespace and punctuation spacing
- **Pipeline** — orchestrates components in a deterministic order

You can customize the pipeline:

```python
from farsflow import (
    Pipeline,
    Normalizer,
    SpaceCleaner,
)

pipeline = Pipeline([
    Normalizer(),
    SpaceCleaner(),
    # JoinerFixer skipped to demonstrate modular behavior
])

text = "می  نويسم  که این   يك   متن  تستي است"
cleaned = pipeline(text)
print(cleaned)
```
Expected output:
```text
می نویسم که این یک متن تستی است
```

## Design Principles

farsflow follows a few core principles:

- deterministic output
- semantic-safe transformations
- opt-in for potentially lossy operations
- zero dependencies
- modular architecture

---

🧪 Testing

```bash
pytest
# or:
pytest path/to/test_file.py
```

---

## 🗺 Roadmap (v0.2.0)

- [ ] configurable preprocessing profiles for different use cases
- [ ] optional emoji and URL cleanup processors
- [ ] compound word normalization (opt-in)
- [ ] additional normalization rules based on real-world corpora
- [ ] performance benchmarking and optimization

---

📄 License

MIT License — see [LICENSE](LICENSE).

---

🤝 Contributing

Contributions are welcome.  
Please open an issue or submit a pull request on GitHub.

📝 Changelog

See [CHANGELOG](CHANGELOG) for version history.
