Metadata-Version: 2.4
Name: urdunorm
Version: 0.1.1
Summary: Urdu text normalization for ASR evaluation, with explicit and reproducible protocols.
Project-URL: Homepage, https://github.com/hunzed/urdunorm
Project-URL: Repository, https://github.com/hunzed/urdunorm
Project-URL: Issues, https://github.com/hunzed/urdunorm/issues
Author: Hunzalah Hassan
License: Apache-2.0
License-File: LICENSE
Keywords: asr,nastaliq,nlp,normalization,unicode,urdu,wer
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: Urdu
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# urdunorm

[![PyPI](https://img.shields.io/pypi/v/urdunorm)](https://pypi.org/project/urdunorm/) [![Python](https://img.shields.io/pypi/pyversions/urdunorm)](https://pypi.org/project/urdunorm/) [![CI](https://github.com/hunzed/urdunorm/actions/workflows/ci.yml/badge.svg)](https://github.com/hunzed/urdunorm/actions/workflows/ci.yml) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21864104.svg)](https://doi.org/10.5281/zenodo.21864104) [![License](https://img.shields.io/pypi/l/urdunorm)](https://github.com/hunzed/urdunorm/blob/main/LICENSE)

Urdu text normalization and WER/CER scoring for ASR evaluation, under named
protocols you can report. Zero runtime dependencies. `pip install urdunorm`

## The problem, in five lines

```python
from urdunorm import wer

reference = ["کے لیے شکریہ"]  # what the transcriber wrote
hypothesis = ["کیلئے شکریہ"]  # what the model emitted, compound joined

wer(reference, hypothesis)  # 0.0
wer(reference, hypothesis, mode="lums-compat")  # 0.667
```

Same audio, same words, one spelling difference, two published protocols. That
gap is why Urdu ASR results do not compare across papers, and why this package
makes you name the protocol.

## What it does to text, under `eval`

| input | output | what happened |
|---|---|---|
| `علي كي` | `علی کی` | Arabic-keyboard letters unified to their Urdu equivalents. Same word, different keyboard. |
| `ﻤﯿﮟ` | `میں` | Presentation forms, common in PDF-extracted text, folded to normal letters. |
| `مُحَمَّد` | `محمد` | Vowel diacritics stripped. Optional in writing, so a model is not charged for omitting them. |
| `۲۰۲۶` | `2026` | All three numeral systems Urdu uses folded onto one. |
| `کیلئے` | `کے لیے` | A compound written without its space, repaired. WER counts whitespace tokens, so this alone costs two errors. |
| `بھائی` | `بھائی` | **Unchanged.** That letter marks aspiration and is phonemic. Merging it into its lookalike reports a *lower* WER by destroying the distinction. |

The last row matters most: phonemic characters are a contract, and the package
refuses to import if a rule table would merge one.

## How it differs from urduhack and LughaatNLP

Those are general Urdu NLP toolkits: tokenizers, POS tagging, NER. This is an
evaluation protocol, normalization plus scoring, nothing else. Different jobs.

Differences are measured, not asserted. This table is generated by
[`compare_normalizers.py --emit-table`](https://github.com/hunzed/urdunorm/blob/main/scripts/compare_normalizers.py)
running each package at a pinned version:

<!-- BEGIN generated comparison -->
| what it is | input | `urdunorm` | `urduhack` | `LughaatNLP` |
|---|---|---|---|---|
| Arabic-keyboard yeh and kaf | `علي كي دكان` | `علی کی دکان` | `علی کی دکان` | `علي كي دكان` |
| presentation forms | `ﻤﻴﮟ` | `میں` | `مےں` | `ﻤﻴﮟ` |
| vowel diacritics | `مُحَمَّد` | `محمد` | `محمّد` | `محمد` |
| Urdu-Indic digits | `۲۰۲۶` | `2026` | `۲۰۲۶` | `۲۰۲۶` |
| joined compound | `کیلئے` | `کے لیے` | `کیلئے` | `کیلئے` |
| aspiration, must not change | `بھائی` | `بھائی` | `بھائی` | `بھائی` |
| alef madda, must not change | `آج` | `آج` | `آج` | `اج` |

Measured on urdunorm 0.1.0, urduhack 1.1.1, LughaatNLP 1.3.1.
<!-- END generated comparison -->

Both also carry heavy dependencies: `urduhack` imports TensorFlow at package
import; `LughaatNLP` requires `torch`, `transformers`, `scikit-learn` and `scipy`.

## Modes

| Mode | Characters | Diacritics | Punctuation | Digits | Space repair | ZWNJ |
|---|---|---|---|---|---|---|
| `eval` | unified | stripped | dropped | unified | on | to space |
| `display` | unified | kept | kept | kept | off | kept |
| `lums-compat` | unified | stripped | 5 chars only | unified | off | to space |

**Use `eval` unless you know why not.** `display` is for showing transcripts to
people. `lums-compat` reproduces
[WER We Stand (COLING 2025)](https://aclanthology.org/2025.coling-main.397/), the
only published Urdu ASR benchmark, so numbers can be compared against it. Its
five-character punctuation class keeps the Urdu full stop: that is theirs, not a
bug here. Report under two protocols; a single-protocol Urdu WER compares to
nothing.

## Scoring

```python
from urdunorm import wer, cer, score, normalize, EVAL

normalize("کیلئے", mode="eval")
wer(references, hypotheses, mode="eval")
cer(references, hypotheses, mode="eval")

result = score(references, hypotheses, mode="eval")
str(result)  # 'WER 12.40% [eval]'
EVAL.with_(repair_spaces=False)  # an ablation is a field override
```

Both sides are normalized under the named mode and the result records which
protocol produced it. Tokenization is a whitespace split. The corpus rate is
total errors over total reference tokens, never the mean of per-utterance rates,
which would weight a three-word clip like a thirty-word one.

## More

[Why this exists](https://github.com/hunzed/urdunorm/blob/main/docs/why.md) ·
[Measured comparison](https://github.com/hunzed/urdunorm/blob/main/docs/normalizer-comparison.md) ·
[Where the rules come from](https://github.com/hunzed/urdunorm/blob/main/docs/grounding.md) ·
[Review the Urdu rules](https://github.com/hunzed/urdunorm/blob/main/docs/data-review.md) (no Python needed) ·
[Contributing](https://github.com/hunzed/urdunorm/blob/main/CONTRIBUTING.md)

The space-repair lexicon and the number tables have not been signed off by a
native speaker, so `numbers_to_words` ships off in every mode.

## Citing

```bibtex
@software{urdunorm,
  title  = {urdunorm: Urdu text normalization for reproducible ASR evaluation},
  author = {Hassan Bhatti, Hunzalah},
  year   = {2026},
  doi    = {10.5281/zenodo.21864104},
  url    = {https://github.com/hunzed/urdunorm}
}
```

Apache-2.0.
