Metadata-Version: 2.4
Name: uztextnorm
Version: 0.3.1
Summary: Uzbek Text Normalization Library for TTS and NLP applications
Project-URL: Homepage, https://gitlab.com/adizbek/uztextnorm
Project-URL: Repository, https://gitlab.com/adizbek/uztextnorm
Project-URL: Issues, https://gitlab.com/adizbek/uztextnorm/-/issues
Author-email: Adizbek Ergashev <adizbek1998@gmail.com>
License: MIT
License-File: LICENSE
Keywords: nlp,normalization,text,transliteration,tts,uzbek
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: black>=23.0.0; extra == 'dev'
Requires-Dist: build>=1.2.1; extra == 'dev'
Requires-Dist: gradio>=6.20.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Requires-Dist: twine>=6.2.0; extra == 'dev'
Description-Content-Type: text/markdown

# UzTextNorm

[![PyPI version](https://img.shields.io/pypi/v/uztextnorm.svg)](https://pypi.org/project/uztextnorm/)
[![Python versions](https://img.shields.io/pypi/pyversions/uztextnorm.svg)](https://pypi.org/project/uztextnorm/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

Uzbek Text Normalization Library for TTS and NLP applications.

## Features

- **Unicode Normalization**: Normalizes text to NFC form
- **Cyrillic to Latin Transliteration**: Automatic conversion of Uzbek Cyrillic to Latin script
- **Number Expansion**: Converts numbers to Uzbek words
  - Integers: `123` → `bir yuz yigirma uch`
  - Decimals: `3.14` → `uch butun o'n to'rt`
  - Fractions: `3/4` → `to'rtdan uch`
  - Currency: `10,000 so'm` → `o'n ming soʻm`
  - Percentages: `15%` → `o'n besh foiz`
- **Punctuation Normalization**: Standardizes punctuation and whitespace
- **URL/Email Preservation**: Protects URLs and emails from normalization
- **Custom Rules**: Support for custom pre/post processing rules

## Installation

```bash
pip install uztextnorm
```

## Usage

### Basic Usage

```python
from uztextnorm import UzbekTextNormalizer

# Create a normalizer instance
normalizer = UzbekTextNormalizer()

# Normalize text
text = "Narx: 10,000 so'm; 15% oshdi"
result = normalizer(text)
print(result)
# Output: "narx: o'n ming soʻm; o'n besh foiz oshdi"
```

### Cyrillic to Latin Transliteration

```python
text = "Ўзбекистон Республикаси"
result = normalizer(text)
print(result)
# Output: "o'zbekiston respublikasi"
```

### Number Expansion

```python
# Integers
normalizer("123")  # "bir yuz yigirma uch"

# Decimals
normalizer("3.14")  # "uch butun o'n to'rt"

# Fractions
normalizer("3/4")  # "to'rtdan uch"
normalizer("1/2")  # "yarim"
normalizer("2 3/4")  # "ikki butun to'rtdan uch"

# Currency
normalizer("10,000 so'm")  # "o'n ming soʻm"
normalizer("2,500 $")  # "ikki ming besh yuz dollar"

# Percentages
normalizer("15%")  # "o'n besh foiz"
```

### Configuration Options

```python
normalizer = UzbekTextNormalizer(
    to_lower=True,  # Convert to lowercase (default: True)
    transliterate_cyrillic=True,  # Transliterate Cyrillic (default: True)
    custom_pre_rules=[],  # Custom pre-processing functions
    custom_post_rules=[]  # Custom post-processing functions
)
```

### Custom Processing Rules

```python
def remove_hashtags(text: str) -> str:
    return text.replace('#', '')

def add_period(text: str) -> str:
    if not text.endswith('.'):
        return text + '.'
    return text

normalizer = UzbekTextNormalizer(
    custom_pre_rules=[remove_hashtags],
    custom_post_rules=[add_period]
)
```

## Module Structure

```
uztextnorm/
├── __init__.py          # Package initialization
├── constants.py         # Constants and language data
├── utils.py            # Utility functions
├── transliterator.py   # Cyrillic-to-Latin conversion
├── tokenizer.py        # Token shielding (URLs, emails)
├── numbers.py          # Number expansion
└── normalizer.py       # Main normalizer class
```

## Requirements

- Python >= 3.10

## Development

Clone the repository and install the development dependencies:

```bash
git clone https://gitlab.com/adizbek/uztextnorm.git
cd uztextnorm
pip install -e ".[dev]"
```

Run the test suite:

```bash
pytest
```

Format and lint:

```bash
black src tests
ruff check src tests
```

Build the distribution artifacts:

```bash
python -m build
```

## Links

- **PyPI**: https://pypi.org/project/uztextnorm/
- **Repository**: https://gitlab.com/adizbek/uztextnorm
- **Issues**: https://gitlab.com/adizbek/uztextnorm/-/issues
- **Changelog**: [CHANGELOG.md](CHANGELOG.md)

## Author

**Adizbek Ergashev** — adizbek1998@gmail.com

## License

MIT License — see [LICENSE](LICENSE) for details.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.
