Metadata-Version: 2.3
Name: polish-whisper-normalizer
Version: 0.1.12
Summary: Polish text normalizer for Whisper ASR output (keeps diacritics, normalizes numbers, time, currency)
Keywords: whisper,asr,polish,normalization,nlp
Author: NeonFeline
Author-email: NeonFeline <maciejlachut77@gmail.com>
License: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: regex
Requires-Dist: morfeusz2
Requires-Dist: jiwer>=3.0 ; extra == 'jiwer'
Requires-Python: >=3.10
Project-URL: Homepage, https://github.com/NeonFeline/polish-whisper-normalizer
Project-URL: Repository, https://github.com/NeonFeline/polish-whisper-normalizer
Project-URL: Issues, https://github.com/NeonFeline/polish-whisper-normalizer/issues
Provides-Extra: jiwer
Description-Content-Type: text/markdown

# Polish Whisper Normalizer

<p align="center">
  <a href="https://github.com/NeonFeline/polish-whisper-normalizer/actions"><img alt="CI" src="https://github.com/NeonFeline/polish-whisper-normalizer/actions/workflows/ci.yml/badge.svg"></a>
  <a href="https://pypi.org/project/polish-whisper-normalizer/"><img alt="PyPI" src="https://img.shields.io/pypi/v/polish-whisper-normalizer.svg"></a>
  <a href="https://www.python.org/downloads/"><img alt="Python" src="https://img.shields.io/badge/python-%3E%3D3.10-blue"></a>
  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-MIT-green"></a>
  <a href="https://neonfeline.github.io/polish-whisper-normalizer/"><img alt="Docs" src="https://img.shields.io/badge/docs-MkDocs-blueviolet"></a>
</p>

> **Polish port of [OpenAI Whisper](https://github.com/openai/whisper) normalizers** — preserves `ąćęłńóśźż`, normalizes numbers, time, currency, dates and declensions via **Morfeusz2**.

[English](#english) | [Polski](#polski)

---

## English

### Features

| Area | Example | Normalized |
|---|---|---|
| **Diacritics** | `Żółć!` | `żółć` (kept) / `zolc` with `remove_diacritics=True` |
| **Cardinal / ordinal** | `sto dwadzieścia trzy`, `dwudziestu pięciu`, `pierwszego → 1` | `123`, `25`, `1` (ordinal dots dropped) |
| **Numeral separation** | `10, 500, 90` vs `dwa tysiące dwadzieścia trzy` | `10 500 90` vs `2023` (punctuation/composition aware) |
| **Time** | `piąta trzydzieści`, `wpół do ósmej`, `o piątej`, `piąta rano`, `od piątej do szóstej` | `5:30`, `7:30`, `o 5:00`, `5:00 rano`, `od 5:00 do 6:00` |
| **Time with `minut`** | `dziesięć minut po piątej`, `za dwadzieścia minut ósma` | `5:10`, `7:40` |
| **Fractions** | `jedna trzecia`, `trzy czwarte` | `1/3`, `3/4` (keeps `/`, idempotent) |
| **Half** | `pół litra`, `półtora`, `dwa i pół` | `0.5 litra`, `1.5`, `2.5` |
| **Currency** | `pięć złotych`, `€10`, `pięć złotówek` | `5 zł`, `10 €`, `5 zł` |
| **Percent** | `pięć procentów` | `5%` |
| **Ordinal multipliers** | `tysiąc dziewięćsetny` | `1900` |
| **Dates** | `5 maja` → `05.05` · `piątego maja` → `05.05` · `piątego maja 2026` → `05.05.2026` · `piątego maja roku dwa tysiące dwudziestego szóstego` → `05.05.2026` | `DD.MM` / `DD.MM.YYYY` zero-padded, conditional (`maja` alone stays `maja`, avoids `Maja`→`5`) |
| **Geographic guard** | `na północ` stays, `jest północ` → `jest 0:00` | no false `0:00` for north |

**Pipeline** (`text.py`): `lower → brackets/parens/ignore → r.→roku → sentence periods/ellipsis → time (Morfeusz) → decimal ,→. → numeral boundaries → remove_symbols(keep=".:/%$€£¢+-") → numbers (Morfeusz) → months (conditional, Morfeusz) → dates → ordinal-dot cleanup`. Configurable via `PolishTextNormalizer(date_format=...)`. Diacritic-less ASR (`czterdziesci`, `piec`, `wpol`) handled via `utils` + `Morfeusz.generate`.

### Installation

```bash
uv sync                 # dev + morfeusz2
uv pip install -e .     # editable
# or from PyPI
uv pip install polish-whisper-normalizer
# with jiwer extra
uv pip install "polish-whisper-normalizer[jiwer]"
```

Requires **Python >=3.10**, `regex`, `morfeusz2`.

### Quickstart

```python
from polish_whisper_normalizer import PolishTextNormalizer, BasicTextNormalizer

n = PolishTextNormalizer()
n("Było piętnaście po piątej, minus dziesięć stopni.")
# → "było 5:15 -10 stopni"

n("Spotkanie dwudziestego pierwszego maja o piętnastej trzydzieści.")
# → "spotkanie 21.05 o 15:30"

n("piątego maja roku dwa tysiące dwudziestego szóstego")
# → "05.05.2026"   # uniform, no trailing "r"

n("5 maja")  # digit day works too
# → "05.05"

n("pięć złotówek, pięć procentów, pół litra, jedna trzecia")
# → "5 zł 5% 0.5 litra 1/3"

# custom date format
PolishTextNormalizer(date_format="%Y-%m-%d")("piątego maja 2026")
# → "2026-05-05"
PolishTextNormalizer(date_format="{day}/{month}/{year}")("piątego maja 2026")
# → "5/5/2026"

# diacritics
BasicTextNormalizer()("Żółć!")  # → "żółć"
BasicTextNormalizer(remove_diacritics=True)("Żółć!")  # → "zolc"
# diacritic-less numbers also work
PolishTextNormalizer()("trzysta czterdziesci osiem")  # → "348"
```

### Jiwer — WER with Polish normalization

```bash
uv pip install "polish-whisper-normalizer[jiwer]"
# or
uv sync --extra jiwer
```

Without normalization `jiwer.wer` penalizes `piątego maja 2026` vs `05.05.2026` as 100% error. With `PolishTransform` they match:

```python
import jiwer
from polish_whisper_normalizer.jiwer import wer, PolishTransform, polish_transform

# helper (recommended) — normalizes both sides with PolishTextNormalizer
wer("piątego maja 2026", "05.05.2026")  # → 0.0
wer("o piątej", "o 5:00")  # → 0.0
wer("pięć złotówek", "5 zł")  # → 0.0

# raw jiwer would be 1.0
jiwer.wer("piątego maja 2026", "05.05.2026")  # → 1.0

# via jiwer transforms (for pipelines)
jiwer.wer("piątego maja 2026", "05.05.2026",
          reference_transform=polish_transform,
          hypothesis_transform=polish_transform)  # → 0.0

# custom date_format is forwarded
wer("piątego maja 2026", "2026-05-05", date_format="%Y-%m-%d")  # → 0.0

# manual Compose
tr = jiwer.Compose([PolishTransform(date_format="%Y-%m-%d"), jiwer.RemoveMultipleSpaces(), jiwer.Strip(), jiwer.ReduceToListOfListOfWords()])
jiwer.wer("piątego maja 2026", "2026-05-05",
          reference_transform=tr, hypothesis_transform=tr)  # → 0.0
```

`PolishTransform` wraps `PolishTextNormalizer` (`date_format` kwarg supported) and returns `str`; `polish_transform` is the ready `Compose` ending with `ReduceToListOfListOfWords` required by `jiwer.wer`.

### Architecture

```
src/polish_whisper_normalizer/
  basic.py        # Whisper basic normalizer, diacritics-aware
  lemmatizer.py   # PolishLemmatizer – thin Morfeusz2 wrapper (analyse/generate)
  utils.py        # strip_diacritics, with_ascii_variants – single place for ASCII fallback
  numbers.py      # PolishNumberNormalizer – cardinals/ordinals/currency/percent
  time.py         # PolishTimeNormalizer – HH:MM, wpół/za/po, geographic guard
  text.py         # PolishTextNormalizer – full pipeline: time → numbers → months → dates
  polish.py       # compatibility re-export shim
  jiwer.py        # PolishTransform / wer helper
```

Declension is never re-implemented: base nominative lexicons are stored, all declined forms are resolved via `PolishLemmatizer.analyse` and `generate` (Morfeusz2). ASCII-folded variants are derived once via `utils` and `Morfeusz.generate`, so diacritic-less ASR needs no duplicated dictionaries.

<details><summary>Components</summary>

```python
from polish_whisper_normalizer import PolishNumberNormalizer, PolishTimeNormalizer, PolishLemmatizer

PolishNumberNormalizer()("dwudziestu pięciu złotych")  # → "25 zł"
PolishTimeNormalizer()("wpół do ósmej")  # → "7:30"
PolishLemmatizer().analyse("dwudziestu")  # → [("dwadzieścia","num")]
```

| Class | Description |
|---|---|
| `PolishTextNormalizer(date_format="{day:02d}.{month:02d}.{year}")` | Full pipeline, `date_format` supports `str.format` (`{day}`, `{month}`, `{year}`) and `strftime` (`%d.%m.%Y`) |
| `PolishNumberNormalizer` | Words → digits, currency/percent/decimals/signs/ordinals |
| `PolishTimeNormalizer` | Spoken time → `HH:MM`, guards `o 5. stronie` vs `o 5:00`, `na północ` vs `0:00` |
| `PolishLemmatizer` | Morfeusz2 wrapper |
| `BasicTextNormalizer` | Whisper `basic` but diacritics-aware |

</details>

### API Docs

Full API: **[neonfeline.github.io/polish-whisper-normalizer](https://neonfeline.github.io/polish-whisper-normalizer/)** (`mkdocs serve` locally).

### Development

```bash
uv sync --group dev
uv run pytest -q          # 600+ tests
uv run mypy src           # strict, py.typed
uv run ruff check src tests && uv run ruff format --check src tests
uv build
mkdocs serve              # strict
```

- `py.typed` + `mypy --strict` (`warn_unused_ignores=false`)
- `ruff` + `pre-commit` + GitHub Actions (`ci.yml`: lint → mypy → pytest --cov → build)
- Validated on **BIGOS v2 + PELCRA** (2397 samples, ~5% number words)

### License

MIT — see `LICENSE` (inherits Whisper MIT for `basic.py`).

---

## Polski

### Funkcje

| Obszar | Przykład | Po normalizacji |
|---|---|---|
| **Znaki diakrytyczne** | `Żółć!` | `żółć` (zachowane) |
| **Liczebniki** | `sto dwadzieścia trzy`, `pierwszego` | `123`, `1.` |
| **Czas** | `piąta trzydzieści`, `o piątej`, `od piątej do szóstej` | `5:30`, `o 5:00`, `od 5:00 do 6:00` |
| **Daty** | `5 maja` → `05.05` · `piątego maja 2026` → `05.05.2026` | `DD.MM` / `DD.MM.RRRR`, warunkowo (`maja` samo → `maja`) |
| **Waluta / procent / ułamki** | `pięć złotówek`, `procentów`, `1/3`, `pół litra` | `5 zł`, `5%`, `1/3`, `0.5 litra` |

**Potok:** `lower → czas → liczby → miesiące (warunkowo) → daty`.

### Instalacja i użycie (PL)

```bash
uv sync                 # dev + morfeusz2
uv pip install polish-whisper-normalizer  # z PyPI
uv pip install "polish-whisper-normalizer[jiwer]"  # z jiwer
```

```python
from polish_whisper_normalizer import PolishTextNormalizer
n = PolishTextNormalizer()  # date_format="{day:02d}.{month:02d}.{year}" domyślnie

n("piątego maja roku dwa tysiące dwudziestego szóstego")
# → "05.05.2026"  # jednolicie, bez "r"

n("5 maja")  # też działa
# → "05.05"

# własny format daty
PolishTextNormalizer(date_format="%Y-%m-%d")("piątego maja 2026")
# → "2026-05-05"

# bez znaków diakrytycznych też działa
n("trzysta czterdziesci osiem")
# → "348"
```

`maja` jako imię `Maja` zostaje `maja` (nie `5`), `na północ` nie staje się `0:00`.

#### Jiwer — WER po polsku

```bash
uv pip install "polish-whisper-normalizer[jiwer]"
```

```python
from polish_whisper_normalizer.jiwer import wer
wer("piątego maja 2026", "05.05.2026")  # → 0.0  (bez normalizacji 1.0)
wer("o piątej", "o 5:00")  # → 0.0
```

### Rozwój / testy

Jak wyżej — `uv run pytest`, `mypy`, `ruff`, `mkdocs serve`.

---

<p align="center"><sub>Made for <a href="https://huggingface.co/datasets/amu-cai/pl-asr-bigos-v2">BIGOS</a> / <a href="https://openai.com/research/whisper">Whisper</a> WER — PRs welcome!</sub></p>
