Metadata-Version: 2.4
Name: lemmauz
Version: 0.1.0
Summary: Deterministic FSM Uzbek lemmatizer
Author: Hikmat Abdoollayev
License-Expression: MIT
Project-URL: Homepage, https://github.com/uzumbek/lemmauz
Keywords: uzbek,nlp,lemmatizer,morphology
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# lemmauz

A deterministic finite-state-machine (FSM) lemmatizer for Uzbek, designed for speed and correctness without neural models or external dependencies.

## Installation

```bash
pip install lemmauz
```

Or from source:

```bash
git clone https://github.com/uzumbek/lemmauz.git
cd lemmauz
pip install -e .
```

## Usage

```python
from lemmauz import fsm_lemmatize

# Bare stem (default)
fsm_lemmatize("kelishilganliklari")        # -> "kelishil"
fsm_lemmatize("o'qiyotganlarimizdan")      # -> "o'qi"
fsm_lemmatize("boshqarmoqda")              # -> "boshqar"

# With -moq infinitive suffix for verbal forms
fsm_lemmatize("ishlayotgan", add_moq=True)  # -> "ishlamoq"
fsm_lemmatize("yuraklarida", add_moq=True)  # -> "yurak" (noun, unchanged)
```

## How It Works

lemmauz processes words through a fixed 7-layer pipeline that strips suffixes right-to-left:

### Architecture

Each layer handles a specific morphological category. The FSM tries the longest suffix first (greedy match), checks structural validity, and moves to the next layer if no match is found.

| Layer | Category | Examples |
|-------|----------|----------|
| 1 | Word-final nominals | `-lik`, `-siz`, `-roq` → `bilimli` → `bilim` |
| 2 | Iterative nominal loop (CASE → POSS_LONG → PLURAL → DAGI → LIK_CHAIN → LOC → POSS_SHORT) | `shaharlarimizdagi` → `shahar` |
| 3 | Participles (with vowel-connective fallback) | `kelayotgan` → `kel`, `tushunmaganligi` → `tushun` |
| 4 | Voice suffixes (passive, reciprocal, frequentative) | `ko'rishmoq` → `ko'rish` |
| 5 | Direct VTM / VPN (verb tense/mood/person) | `yaptim`, `moqda`, `di` |
| 6 | Phonological restoration | `lig`→`lik`, `g'`→`q` |
| 7 | Mutation dictionary fallback | `shahr`→`shahar`, `yurag`→`yurak` |

### Key Design Decisions

- **No normalization of digraphs**: `sh` → `ş` and `ch` → `ç` are intentionally avoided. Uzbek orthography treats these as two-letter sequences, and normalizing them corrupts stems like `do'st`.
- **Phoneme-aware MIN_STEM**: The minimum stem length check operates on phonemes (after normalizing `o'` → single char, `g'` → single char), not raw bytes. This correctly counts multi-character sequences as one unit.
- **Vowel-connective fallback in participle layer**: When a vowel-leading participle suffix (e.g., `ayotgan`) matches but leaves an unnatural consonant cluster, the FSM tries the shorter variant (`yotgan`) to preserve what's likely a stem-final vowel. This handles cases like `ishla + yotgan` → surface form `ishlayotgan`.
- **Mutation dictionary**: A small lookup table (~20 entries) restores stems that undergo predictable phonological changes before consonant-initial suffixes:
  - Consonant mutation: `k` → `g` (`yurak` → `yurag-`), `q` → `g'` (`quloq` → `qulog'-`)
  - Vowel dropping: `a/i` → ∅ (`shahar` → `shahr-`, `burun` → `burn-`)
  - Applied as a post-processing fallback after all structural layers.

## Benchmark Results

### Main benchmark — 120 complex agglutinative chains

Deeply suffixed words with 5+ morphological layers:

| Lemmatizer | Correct | Total | Accuracy |
|------------|---------|-------|----------|
| **lemmauz** | 120 | 120 | **100.0%** |
| UzMorphAnalyser | 79 | 120 | 65.8% |
| UzbekLemma | 45 | 120 | 37.5% |

lemmauz outperforms existing open-source lemmatizers by a wide margin on complex chains, primarily because:

- **UzMorphAnalyser** leaves participle suffixes attached and corrupts apostrophe-containing stems (`o'` → `o?`)
- **UzbekLemma** over-applies `-moq` to nouns and fails to strip deep nominal chains

### Exceptional words — 163 edge cases

Words with phonological mutations, ambiguous suffixes, and irregular patterns:

| Lemmatizer | Correct | Total | Accuracy |
|------------|---------|-------|----------|
| UzMorphAnalyser | 80 | 163 | **49.1%** |
| **lemmauz** | 73 | 163 | 44.8% |
| UzbekLemma | 71 | 163 | 43.6% |

All three lemmatizers struggle on this set. UzMorphAnalyser leads slightly, but at the cost of aggressive over-stripping Arabic loan roots (`muhim` → `muh`, `iqlim` → `iql`). lemmauz trades a small accuracy loss for correct preservation of known loans via a vowel-count guard. Failures fall into two categories (see "What Doesn't Work Yet"):

- **Arabic loan ambiguity (`-im`)**: 90 misses — stem-final `-im` vs. possessive suffix
- **Vowel-mutating stems**: 75 misses — vowel dropping/assimilation before consonant-initial suffixes

## What Doesn't Work Yet

### Arabic loan words ending in `-im` (90 misses)

The hardest remaining class involves words ending in `-im`, which is ambiguous between:

1. **Arabic loan roots** where `-im` is part of the stem: `tizim` (system), `iqlim` (climate), `muhim` (important), `taslim` (surrender)
2. **Possessive suffixes** on native stems: `o'g'il + im` → `o'g'lim` (my son), `yurak + im` → `yuragim` (my heart), `sabir + im` → `sabrim` (my patience)

Current behavior uses a vowel-count guard to protect Arabic loans with ≤1 vowel in the remaining stem. This correctly preserves `tizim`, `muhim`, etc., but still produces incorrect results on words like:
- `shonim` → expected `shon`, got `shonim` (Arabic loan preserved, but should strip possessive)
- `sabrim` → expected `sabr`, got `sabrim` (possessive not stripped)

A full solution requires either a dictionary of known Arabic loans or morphological analysis that distinguishes stem-final `-im` from suffix `-im`.

### Possessive suffixes on vowel-mutating stems (75 misses)

When possessive suffixes attach to stems ending in vowels, the stem-final vowel often drops or assimilates:
- `o'g'il + ing` → `o'g'ling` (expected stem `o'g'il`, got `o'g'l`)
- `shahar + i` → `shahri` (expected `shahar`, got `shahr`)

The mutation dictionary (layer 7) covers ~15 common stems, but a complete solution would need either:
- A larger mutation table covering all vowel-mutating stems
- Or structural rules that reinsert the dropped vowel when the FSM result ends in an unnatural consonant cluster

### Short words and edge cases

Words like `orqaside` (expected `orqa`), `kelma` (expected `kelmoq`), and proper nouns (`Azim`) remain challenging due to their brevity or non-standard suffix patterns.

## Project Structure

```
lemmauz/
├── lemmauz/
│   ├── __init__.py        # Public API: fsm_lemmatize()
│   └── lemmauz.py         # Core FSM implementation (single-file, no dependencies)
├── tests/
│   ├── test_without_moq.txt  # Benchmark dataset (120 pairs)
│   └── test_exceptional.txt  # Exceptional words dataset (163 pairs)
├── benchmark.py           # Fair comparison script against UzMorphAnalyser and UzbekLemma
└── pyproject.toml         # Package configuration
```

## Running Benchmarks

```bash
pip install -e .
python benchmark.py
```

## License

MIT
