Metadata-Version: 2.4
Name: urdu-text-eval
Version: 1.0
Summary: Urdu text evaluation & benchmarking library with configurable orthographic normalization (WER, CER, chrF, BLEU).
Author: urdu-text-eval
License: MIT
Project-URL: Homepage, https://github.com/local/urdu-text-eval
Keywords: urdu,evaluation,benchmarking,wer,cer,chrf,bleu,nlp,ocr,asr,transliteration,text-evaluation
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: jiwer>=3.0.0
Requires-Dist: sacrebleu>=2.4.0
Requires-Dist: rapidfuzz>=3.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"

# urdu-text-eval

[![PyPI Version](https://img.shields.io/pypi/v/urdu-text-eval.svg)](https://pypi.org/project/urdu-text-eval/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python Version](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/downloads/)

**urdu-text-eval** is a Python benchmarking library for **Urdu text evaluation** (OCR, ASR, transliteration, machine translation, LLM text generation, and text processing).

It computes standard NLP metrics (**WER, CER, chrF, BLEU, Exact Match, Character Similarity**) with configurable **Urdu orthographic and taxonomy normalization** (handling Arabic lookalike codepoints, zabar/zeer/pesh diacritics, punctuation, numbers, and presentation forms).

---

## Key Features

- **Standard Metrics**: Word Error Rate (WER), Character Error Rate (CER), Word/Char Accuracy, chrF, BLEU, Exact Match, and Levenshtein Character Similarity.
- **Urdu Taxonomy Normalization**: Standardizes Arabic variants (`ك` → `ک`, `ي` → `ی`, `ه` → `ہ`), presentation forms (`ﺍ`, `ﺎ`, `ﺐ`), and combined characters (`ا`+`ٓ` → `آ`).
- **Fine-Grained Diacritic Controls**: Selective removal of اعراب (**Zabar** `َ`, **Zeer** `ِ`, **Pesh** `ُ`, **Tanween** `ً ٌ ٍ`, **Shadda** `ّ`, **Sukun** `ْ`, Maddah, and Hamza marks).
- **Flexible Input Formats**: Evaluates simple list-of-dicts `[{"actual": "...", "pred": "..."}]` with support for common key aliases (`ref`, `target`, `gold`, `Urdu`, `hyp`, `prediction`, `output`).
- **Dual Reporting**: Always reports both **taxonomy-normalized** (primary) and **raw surface** metrics side-by-side.

---

## Installation

Install via pip:

```bash
pip install urdu-text-eval
```

*(Dependencies: `jiwer`, `sacrebleu`, `rapidfuzz`)*

---

## Quick Start

```python
from urdu_text_eval import evaluate

pairs = [
    {"actual": "کیا یہ ہے؟", "pred": "كيا يہ ہے؟"},
    {"actual": "آہ جو دل سے", "pred": "آه جو دل سے"},
]

result = evaluate(pairs)

print(result["summary"])
print(f"WER: {result['wer']:.4f} | CER: {result['cer']:.4f} | chrF: {result['chrf']:.4f}")
```

---

## Output Structure

Calling `evaluate(pairs, per_sample=True)` returns a dictionary containing:

| Key | Type | Description |
|-----|------|-------------|
| `wer` | `float` | Primary Word Error Rate (0.0 = perfect match) |
| `cer` | `float` | Primary Character Error Rate |
| `word_accuracy` | `float` | `1.0 - wer` |
| `char_accuracy` | `float` | `1.0 - cer` |
| `normalized_exact_match` | `float` | Ratio of exact matches after normalization |
| `chrf` | `float` | Character n-gram F-score (sacrebleu) |
| `bleu` | `float` | Corpus BLEU score |
| `mean_char_similarity` | `float` | Mean normalized Levenshtein similarity |
| `raw_wer` / `raw_cer` | `float` | Metrics calculated on raw strings without normalization |
| `raw_exact_match` | `float` | Exact match ratio on raw strings |
| `summary` | `str` | Pre-formatted, printable evaluation report |
| `samples` | `list` | *(Optional, if `per_sample=True`)* List of dicts with per-row scores and normalized strings |

---

## Normalization & Customization

Urdu text often varies in orthography (e.g. Arabic vs. Urdu keyboards, presence of diacritics/اعراب, punctuation). You can control normalization behavior precisely:

### 1. Master On/Off

```python
# Default: Normalization ON (Fair orthographic evaluation)
result = evaluate(pairs)

# Raw evaluation (No normalization applied)
result = evaluate(pairs, normalize=False)
```

### 2. Convenience Overrides

You can pass boolean flags directly to `evaluate()`:

```python
result = evaluate(
    pairs,
    taxonomy=True,            # Convert Arabic codepoints (ك/ي/ه) to Urdu (ک/ی/ہ)
    remove_diacritics=True,   # Remove all اعراب (zabar, zeer, pesh, tanween, etc.)
    remove_zabar=True,        # Remove zabar (َ) only
    remove_zeer=True,         # Remove zeer (ِ) only
    remove_pesh=True,         # Remove pesh (ُ) only
    remove_punctuation=True,  # Remove Urdu (؛،؟۔٪) and ASCII punctuation
    remove_digits=False,      # Remove ASCII and Urdu digits
)
```

### 3. Using `NormalizeConfig` or Presets

For reusability across benchmarks, configure a `NormalizeConfig`:

```python
from urdu_text_eval import evaluate, NormalizeConfig

# Presets
cfg_default = NormalizeConfig.default()          # Taxonomy + diacritics + whitespace
cfg_raw     = NormalizeConfig.none()             # Raw string comparison
cfg_full    = NormalizeConfig.full()             # Everything on (including punctuation & digit stripping)
cfg_diac    = NormalizeConfig.diacritics_only()  # Only remove اعراب
cfg_punct   = NormalizeConfig.punctuation_only() # Only remove punctuation
cfg_tax     = NormalizeConfig.taxonomy_only()    # Only taxonomy mapping

# Custom Configuration
cfg = NormalizeConfig(
    enabled=True,
    taxonomy=True,             # Map Arabic/presentation characters
    combine_characters=True,   # Combine characters (ا + ٓ → آ)
    remove_zabar=True,         # Remove zabar
    remove_zeer=True,          # Remove zeer
    remove_pesh=True,          # Remove pesh
    remove_tanween=True,       # Remove tanween
    remove_shadda=True,        # Remove shadda
    remove_sukun=True,         # Remove sukun
    remove_punctuation=False,  # Keep punctuation
    remove_digits=False,       # Keep digits
)

result = evaluate(pairs, config=cfg)
```

---

## Standalone Normalizer Function

You can also use the Urdu normalizer directly on individual text strings:

```python
from urdu_text_eval import normalize_urdu, NormalizeConfig

# Default normalization
clean_text = normalize_urdu("كيا يہ ہے؟")
# Output: "کیا یہ ہے؟"

# Remove diacritics (zabar/zeer/pesh) only
clean_text = normalize_urdu("شیرِ پنجاب", remove_diacritics_all=True)
# Output: "شیر پنجاب"

# Punctuation removal
clean_text = normalize_urdu("سلام، دنیا!", config=NormalizeConfig.punctuation_only())
# Output: "سلام دنیا"
```

---

## API Summary

```python
from urdu_text_eval import (
    evaluate,           # Core benchmark function: evaluate([{"actual": "...", "pred": "..."}])
    NormalizeConfig,    # Configuration dataclass & presets for text normalization
    normalize_urdu,     # Single-string Urdu normalization function
    compute_metrics,    # Lower-level function: compute_metrics(references, hypotheses)
    format_metrics,     # Formats metrics dict into a human-readable text summary
    per_sample_errors,  # Returns list of per-item error breakdown dicts
)
```

---

## License

Distributed under the [MIT License](https://opensource.org/licenses/MIT).
