Metadata-Version: 2.4
Name: JT_script_normaliser
Version: 0.1.0
Summary: Language-aware text normalisation for 22 Indian and several foreign languages
Author-email: Manas Dhir <manas@joshtalks.com>
License-Expression: MIT
License-File: LICENSE
Keywords: asr,indic,nlp,normalisation,normalization,text,unicode
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Bengali
Classifier: Natural Language :: Hindi
Classifier: Natural Language :: Tamil
Classifier: Natural Language :: Telugu
Classifier: Natural Language :: Urdu
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Provides-Extra: ftfy
Requires-Dist: ftfy>=6; extra == 'ftfy'
Description-Content-Type: text/markdown

# JT_script_normaliser

Language-aware text normalisation for the 22 scheduled Indian languages plus
several foreign languages (English, Romanian, Portuguese, Vietnamese, Arabic).

It is built for ASR / speech pipelines: it canonicalises Unicode, strips or
preserves punctuation and symbols depending on the mode, normalises digit
scripts, and flags text that contains characters foreign to the chosen
language.

## Install

```bash
pip install JT_script_normaliser
```

Optional mojibake / encoding repair via [ftfy](https://ftfy.readthedocs.io):

```bash
pip install "JT_script_normaliser[ftfy]"
```

## Quick start

```python
from JT_script_normaliser import verbatim, verbatim_with_numbers, display

verbatim("नमस्ते, दुनिया!", "hindi")
# 'नमस्ते दुनिया'

verbatim_with_numbers("मेरे पास 5 आम", "hindi", number_script="native")
# 'मेरे पास ५ आम'

display("कीमत ₹1,00,000.50 है।", "hindi")
# 'कीमत ₹100000.50 है'
```

## Modes

All modes apply a shared base: Unicode NFC, whitespace collapsing, removal of
invisible formatting characters (BOM, bidi marks, ...), language-specific
canonicalisation (e.g. Malayalam chillu folding), and a **foreign-character
gate** (see below).

| Mode | Punctuation | Symbols (₹ % °) | Numbers |
|------|-------------|-----------------|---------|
| `verbatim` | removed | removed | removed (presence redacts) |
| `verbatim_with_numbers` | removed | removed | kept, normalised |
| `display` | removed | **kept** | kept, normalised |

### `verbatim(text, language, *, remove_zero_width=True, use_ftfy=False)`
Spoken text only — punctuation, symbols and numbers are stripped, meaning
preserved. If any number or foreign character remains, the whole input is
redacted (see below).

### `verbatim_with_numbers(text, language, *, number_script="latin", remove_zero_width=True, use_ftfy=False)`
Like `verbatim`, but numbers are **kept and unified** to one digit script.
A decimal point between two digits is kept (`3.14`); a grouping comma between
two digits is removed (`1,00,000` -> `100000`).

### `display(text, language, *, number_script="latin", remove_zero_width=True, use_ftfy=False)`
Readable output for downstream/ASR consumers. Keeps numbers, decimals and
**essential symbols** while removing ordinary punctuation:

- Always kept: currency (`$ ₹ €`), `° © ™ ®`, and `% ‰ ‱ @ # &`.
- Kept only next to a digit: math symbols (`+ − × ÷ = ± < > | ~`) and the
  affixes `: - / * ^` (so `5:30`, `-9`, `3/4`, `2*3` are kept, but `a-b`,
  `and/or`, `x = y` are not).
- Vulgar fractions are folded to ASCII: `½` -> `1/2`.

```python
display("meet at 5:30, pay $5 + €3 = ¥8 - well-known", "english")
# 'meet at 5:30 pay $5 + €3 = ¥8 well known'
```

## Options

| Option | Default | Values | Applies to |
|--------|---------|--------|------------|
| `language` | — (required) | name or alias / ISO code (`"hindi"`, `"hi"`) | all |
| `number_script` | `"latin"` | `"latin"`, `"native"` | numbers, display |
| `remove_zero_width` | `True` | `True`, `False` | all |
| `use_ftfy` | `False` | `True`, `False` (needs `[ftfy]` extra) | all |

- **`number_script`** — `"latin"` renders all digits as `0-9`; `"native"`
  renders them in the language's script (e.g. Devanagari `५`, Tamil `௫`,
  Urdu `۵`). Both Latin and native input digits are unified to the target.
- **`remove_zero_width`** — ZWJ/ZWNJ are removed by default. Set `False` to
  preserve them (e.g. to keep Telugu vibhakti rendering). Lexically meaningful
  sequences (Malayalam chillu, Bengali khanda-ta) are folded to their atomic
  forms first, so they survive either way.
- **`use_ftfy`** — when `True`, runs `ftfy.fix_text` first to repair mojibake
  (e.g. `à¤¹à¤¿...` -> `हि...`).

## Foreign-character gate

After normalisation, any surviving character that is not a letter/mark of the
chosen language — a foreign letter, or any digit in `verbatim` mode — causes the
whole input to be returned as a redaction marker:

```python
verbatim("नमस्ते hello", "hindi")
# 'REDACTED_FOREIGN_CHAR (h)'

verbatim_with_numbers("मेरे पास ৫ आम", "hindi")   # Bengali digit in Hindi
# 'REDACTED_FOREIGN_CHAR (৫)'
```

## Supported languages

```python
from JT_script_normaliser import available_languages
available_languages()
```

**Indian:** assamese, bengali, bodo, bhojpuri, chhattisgarhi, dogri, gujarati,
hindi, kannada, kashmiri, konkani, maithili, malayalam, manipuri (Meetei
Mayek), marathi, nepali, odia, punjabi, sanskrit, santali, sindhi, tamil,
telugu, urdu.

**Foreign:** english, romanian, portuguese, vietnamese, arabic.

Common aliases and ISO codes are accepted (`hi`, `bn`, `ta`, `ur`, `en`, ...).

## Per-language canonicalisation

Beyond the base pipeline, some languages get script-specific fixes:

- **Malayalam** — legacy chillu (consonant + virama + ZWJ) folded to atomic
  chillu (`ൻ ർ ൽ ...`), so ZWJ removal doesn't corrupt them.
- **Bengali / Assamese** — legacy khanda-ta folded to atomic `ৎ`.
- **Urdu / Sindhi / Kashmiri / Arabic** — presentation forms folded to base
  letters; tatweel (kashida) removed.
- **Romanian** — cedilla `ş ţ` folded to comma-below `ș ț`.

## License

MIT