Metadata-Version: 2.4
Name: urdunorm
Version: 0.1.0
Summary: Urdu text normalization for ASR evaluation, with explicit and reproducible protocols.
Project-URL: Homepage, https://github.com/hunzed/urdunorm
Project-URL: Repository, https://github.com/hunzed/urdunorm
Project-URL: Issues, https://github.com/hunzed/urdunorm/issues
Author: Hunzalah Hassan
License: Apache-2.0
License-File: LICENSE
Keywords: asr,nastaliq,nlp,normalization,unicode,urdu,wer
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: Urdu
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# urdunorm

Urdu text normalization for ASR evaluation, with explicit and reproducible
protocols.

```bash
pip install urdunorm
```

```python
from urdunorm import normalize

normalize("کیلئے", mode="eval")  # 'کے لیے'
normalize("بھائی", mode="eval")  # 'بھائی'   aspiration preserved
normalize("ہاں", mode="eval")  # 'ہاں'     nasalization preserved
```

```bash
urdunorm --mode eval < ref.txt > ref.norm.txt
urdunorm --mode lums-compat --jsonl-field text < hyp.jsonl
```

## Why this exists

Urdu word error rate is not currently a well-defined quantity. Published numbers
for the same model on the same benchmark differ by tens of points across studies.
Most of that spread is not modelling. It comes from two choices that papers rarely
state: which held-out set was used, and how the text was normalized before
scoring.

Urdu makes the normalization choice unusually consequential.

The script has near-duplicate characters that carry no phonemic contrast, so
`ي` U+064A and `ی` U+06CC are the same letter typed on different
keyboards. A normalizer that ignores this charges a model for a keyboard layout.

It also has near-duplicate characters that carry a real contrast, and merging them
is the standard bug. `ھ` U+06BE marks aspiration: `بھائی` is
"brother", `بہائی` is not the same word. `ں` U+06BA marks
nasalization: `ہاں` is "yes", `ہان` is not a word. A normalizer
that folds either of these into its lookalike reports a lower WER by destroying
information.

And the same phrase is routinely written with or without an internal space.
`کیلئے` and `کے لیے` are the same two words. WER counts
whitespace-separated tokens, so that spelling difference alone scores as two
errors. This is where most of the gap between published numbers lives.

## Scoring

`wer` and `cer` take a string or a list of strings, normalize both sides under a
named mode, and pool the result over the corpus.

```python
from urdunorm import wer, cer, score

wer(references, hypotheses, mode="eval")
cer(references, hypotheses, mode="eval")

result = score(references, hypotheses, mode="eval")
result.rate, result.substitutions, result.deletions, result.insertions
str(result)  # 'WER 12.40% [eval]'
```

Tokenization is a whitespace split, and the corpus rate is total errors over
total reference tokens, never the mean of per-utterance rates. Both are stated
here because leaving them to the caller is how the same model ends up with two
different published numbers.

There is no dependency on a WER library. Edit distance is thirty lines, and
inheriting another package's default transform would reintroduce exactly the
unstated normalization this one removes. The test suite checks agreement with
`jiwer` where it is installed: WER, CER, and total edit distance match exactly.
The substitution/deletion/insertion split does not always match, because edit
distance has several equally optimal alignments and the split is a tie-breaking
choice rather than a fact.

## Modes

| Mode | Characters | Diacritics | Punctuation | Digits | Space repair | Latin |
|---|---|---|---|---|---|---|
| `eval` | unified | stripped | dropped | unified to ASCII | on | lowercased |
| `display` | unified | kept | kept | kept | off | kept |
| `lums-compat` | unified | stripped | dropped | unified to ASCII | off | lowercased |

`eval` is the protocol this project reports under.

`display` is for human-facing transcripts. It fixes keyboard variants, removes
tatweel and zero-width controls, and folds `أ إ ٱ` to plain `ا`. It keeps
diacritics, punctuation, digits and case.

`lums-compat` reproduces the preprocessing of WER We Stand (COLING 2025), the
only published Urdu ASR benchmark, so numbers can be compared against it. Their
scoring code is published; one data file it loads is not, so the character
whitelist here is a reconstruction, validated by feeding it to their own
unmodified scorer and recovering their published table to a mean absolute error
of 0.046 points. `urdunorm.lums` carries the faithful asymmetric reproduction,
and its docstrings record what the paper leaves ambiguous and what we assumed.

Every number this project reports appears twice, once under `eval` and once under
`lums-compat`. A single-protocol Urdu WER is not a comparable quantity.

## What is preserved

These are phonemic in Urdu. No mode merges them, and the package refuses to
import if a rule table would:

`ھ` do-chashmi heh, `ں` noon ghunna, `ے` barree yeh,
`ۓ`, `آ` alef madda, `ئ`, `ؤ`, `ۂ`, and the
retroflex and Perso-Urdu consonants `ٹ ڈ ڑ پ چ ژ گ`.

## Protocols are data

A mode is a frozen dataclass, not a code path. That makes ablations cheap and
makes the exact configuration reportable.

```python
from urdunorm import EVAL, apply_protocol

no_repair = EVAL.with_(repair_spaces=False)
apply_protocol(text, no_repair)
```

```bash
urdunorm --show-protocol --mode eval
```

## Rules live in reviewable data files

The space-repair lexicon and the Urdu number tables are tab-separated files under
`src/urdunorm/data/`, not Python literals, so a native Urdu speaker can review
them without reading code. See [docs/data-review.md](docs/data-review.md).

The number tables and the space-repair lexicon have **not yet been reviewed by a
native speaker**. `numbers_to_words` is off in every mode until they are.

## Dependencies

None at runtime, deliberately. `urdunorm` must stay installable by people who
only want the normalizer, so it never imports `torch`, `transformers`, or
`datasets`.

## Citing

Not yet published. A DOI will be minted at the first tagged release.

## License

Apache-2.0.
