Metadata-Version: 2.5
Name: text-iron
Version: 1.1.1
Summary: Dependency-free Persian/multilingual text normalization for search and matching.
Author: Hamid Asgari
License-Expression: MIT
License-File: LICENSE
Keywords: arabic,farsi,normalization,persian,search,text-processing,unicode
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Persian
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# text-iron

Dependency-free Persian/multilingual text normalization for search and matching
(queries, titles, body content). One implementation, pinned by `NORM_VERSION`, so
text normalized at different times stays **byte-identical** — drift silently
breaks matching.

## Use

```python
from text_iron import normalize_query, normalize_title, normalize_content

normalize_query("Café كيك ۱۲۳")        # -> "cafe کیک 123"
normalize_title("کتاب: جلد ۲")          # -> "کتاب: جلد 2"
normalize_content("بند اول\n\nبند دوم")  # -> keeps the paragraph break
```

The output is a **matching / embedding key**, not display text. Always keep the
original around for display and snippets.

## URLs

`text_iron.url` is a URL normalizer for whatever keys URLs — a crawler's seen-set, an index,
a store of search results — so that everything meets on one key:

```python
from text_iron.url import normalize_url, URL_NORMALIZATION_VERSION

normalize_url("HTTP://Example.IR:80/a/./b?utm_source=x&b=2&a=1#top").value
# -> "http://example.ir/a/b?a=1&b=2"
```

It is not text normalization and uses none of it: a URL is an identifier, and folding its
letters or digits would change it. Its rules are versioned by `URL_NORMALIZATION_VERSION`,
separately from `NORM_VERSION`.

## Pipelines

| function | scripts kept | punctuation | whitespace |
|---|---|---|---|
| `normalize_query` | Persian/Arabic + Latin | none (letters+digits key) | flattened to one line |
| `normalize_title` | all | sentence punctuation | flattened to one line |
| `normalize_content` | all | sentence punctuation | **line/paragraph structure preserved** |

All three share one folding core and are idempotent.

## Components

Composable, single-purpose, pure: `to_unicode_form`, `fold_persian_letters`,
`strip_arabic_diacritics`, `fold_latin_diacritics`, `normalize_digits`,
`to_lowercase`, `canonicalize_zwnj`, `filter_charset` (via `CharPolicy`),
`flatten_whitespace`, `normalize_content_whitespace`, `fingerprint`. Build a new
pipeline by composing these, then pin a new `NORM_VERSION`.

## Design decisions worth knowing

- **Homophones (`س/ص/ث`, `ز/ذ/ض/ظ`, …) are NOT folded** — they are distinct
  letters and real spelling signal.
- **Hamza forms (`أ/إ/ؤ/ئ`), `ة`, `ۀ` are NOT folded** — conservative default;
  folding raises recall. Flip in `fold_persian_letters` / the table if desired.
- **NFKC** is used, which rewrites `m²→m2`, `½→1⁄2`, `①→1`. Switch a pipeline's
  `_fold(..., unicode_form="NFC")` if a use case must keep those glyphs.
- **Query drops non-Persian/Latin scripts; content keeps every script.** The
  charset policy is the main knob (`CharPolicy`).

## Install

```bash
pip install text-iron
```

## Test

```bash
uv run pytest tests            # or:
python tests/test_text_iron.py
```
