Metadata-Version: 2.4
Name: hindi-normalize
Version: 0.1.0
Summary: Convert Hindi number words to digits and normalize Devanagari ASR/OCR text
Project-URL: Homepage, https://github.com/Tushar-9802/hindi-normalize
Project-URL: Issues, https://github.com/Tushar-9802/hindi-normalize/issues
Author-email: Tushar Jaju <tusharbrisingr9802@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Tushar Jaju
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: asr,devanagari,hindi,indic,nlp,normalization,number-words,speech-to-text,text-processing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: Hindi
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# hindi-normalize

Convert Hindi number words to digits and normalize Devanagari text from ASR, OCR, or copy-paste. Handles the encoding variants, punctuation quirks, and number-word spellings that break search, equality checks, tokenization, and downstream extraction. Zero runtime dependencies — pure standard library.

I built this because [Sakhi](https://github.com/Tushar-9802/Sakhi) — an offline medical Hindi voice-to-form tool meant for ASHAs— kept reinventing the same regex passes against Whisper output: number words that never became digits, `५` and `5` treated as different characters, `|` where a danda `।` belonged, the same word encoded two different ways. This is that pass, extracted and generalized.

## Install

```bash
pip install hindi-normalize
```

## Quick start

```python
from hindi_normalize import normalize_transcript

normalize_transcript("आपका BP एक सौ दस बटा सत्तर है, वजन अट्ठावन kg")
# 'आपका BP 110/70 है, वजन 58 kg'
```

`normalize_transcript` runs the whole pipeline. The individual transforms are also exported and usable on their own.

## Numbers

Hindi number words → digits, tolerant of ASR spelling variants (`पाँच`/`पांच`/`पाच`, `सत्तर`/`सतर`). Adjacent unrelated numbers are not summed — `दो तीन दिन` ("two-three days") stays `2 3 दिन`, never `5`.

```python
from hindi_normalize import convert_numbers, parse_hindi_number

convert_numbers("एक सौ दस बटा सत्तर")   # '110 बटा 70'
parse_hindi_number("नौ सौ निन्यानवे")    # 999
```

## Devanagari

`normalize_devanagari` folds the character-level variants OCR/ASR/copy-paste introduce, following the operations the [Indic NLP Library](https://github.com/anoopkunchukuttan/indic_nlp_library) established as standard.

```python
from hindi_normalize import normalize_devanagari

normalize_devanagari("वजन ५८ किलो | ठीक है")   # 'वजन 58 किलो। ठीक है'
```

Default (lossless, on):

- **Invisibles** — strip ZWJ/ZWNJ/ZWSP, BOM, soft hyphen, word joiner; no-break space → space
- **NFC** — unify nukta encodings (`क़` as one codepoint vs `क` + nukta) and matra order
- **Digits** — Devanagari digits (`५`) → ASCII (`5`)
- **Danda** — `|` → `।`, `||` → `॥`, collapse runs, drop space before a danda

Opt-in (lossy, off):

- `remove_nukta` — `क़` → `क`, `ज़` → `ज`
- `nasals` — fold nasal clusters to anusvara (`हिन्दी` → `हिंदी`)
- `chandrabindu` — `ँ` → `ं`
- `visarga` — ASCII colon after a Devanagari letter → visarga (`दु:ख` → `दुःख`)

Unlike a naive `unicodedata`-category filter, nothing here strips the Unicode Mark category, so matras and the virama survive — the [failure mode](https://kavyamanohar.com/post/indic-normalizer/) that silently deletes vowel signs from Indic text.

## Terms

Map spoken/mis-recognized terms to canonical forms. Ships a maternal/child-health (ASHA home-visit) dictionary; pass your own for any domain.

```python
from hindi_normalize import replace_terms, MEDICAL_TERMS

replace_terms("बीबी एक सौ दस बटा सत्तर", MEDICAL_TERMS)   # 'BP एक सौ दस / सत्तर'
replace_terms("क ख", {"क": "K", "ख": "KH"})              # 'K KH'
```

## Repetition

Collapse the back-to-back stutter recognisers emit.

```python
from hindi_normalize import collapse_repetition

collapse_repetition("ठीकठीकठीकठीक है")   # 'ठीक है'
```

For runaway *tail* loops in long LLM generations (repeated sentences/paragraphs) and for general invisible/typographic cleanup of non-Devanagari text, compose with [llmclean](https://github.com/Tushar-9802/llmclean).

## API

| Function | Purpose |
|---|---|
| `normalize_transcript(text, *, terms, numbers, devanagari, repetition, sentence_breaks)` | Full pipeline |
| `convert_numbers(text)` | Hindi number words → digits |
| `parse_hindi_number(text)` | Parse one number expression → `int` |
| `normalize_devanagari(text, *, ...)` | Character-level normalization |
| `strip_zero_width(text)` | Remove invisibles; NBSP → space |
| `devanagari_digits_to_ascii(text)` / `ascii_digits_to_devanagari(text)` | Digit conversion |
| `normalize_danda(text)` | Danda punctuation |
| `fold_nasals(text, *, chandrabindu)` | Nasal clusters → anusvara |
| `replace_terms(text, terms)` | Term dictionary replacement |
| `collapse_repetition(text, *, min_repeats)` | Collapse inline stutter |

Every function is guarded against non-string input and returns the input unchanged rather than raising.

## License

MIT
