Metadata-Version: 2.4
Name: ZirhUzMorphBPE
Version: 0.1.1
Summary: Morphology-aware BPE tokenizer for Uzbek (MorphBPE): words are split into morphemes with ZirhUzMorph, BPE merges never cross morpheme boundaries. Pretrained 32k vocab included.
Author: Xojiakbar Jamoldinov
License: MIT
Project-URL: Homepage, https://github.com/XojiakbarJamoldinov/ZirhUzMorphBPE
Keywords: uzbek,tokenizer,bpe,morphbpe,morphology,nlp,llm
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: ZirhUzMorph>=0.1.1
Requires-Dist: tokenizers>=0.13
Dynamic: license-file

# ZirhUzMorphBPE

**MorphBPE — o'zbek tili uchun morfologik BPE tokenizator.** Har so'z avval
[ZirhUzMorph](https://pypi.org/project/ZirhUzMorph/) bilan morfemalarga ajratiladi, BPE faqat morfema ichida
merge qiladi — `lar`, `da`, `gan`, `imiz` doim bir xil token, asoslar butun qoladi.

```bash
pip install ZirhUzMorphBPE
```

```python
from zirhuzmorphbpe import MorphBPE

tok = MorphBPE()                                   # paket bilan kelgan 32k model
tok.tokens("Kitoblarimizdan biri yöqoldi.")
# ['▁Kitob', 'lar', 'imiz', 'dan', '▁bir', 'i', '▁yöqol', 'di', '.']
enc = tok.encode("Teatrlaştirilgan asarlar Toşkentda körsatildi.")
enc.ids                                            # model uchun ID lar
tok.decode(enc.ids)                                # asl matn — yo'qotishsiz
```

Terminalda: `zirhuzmorphbpe encode "Kitoblarimizdan biri yöqoldi."`

## O'z korpusingizda o'rgatish
```bash
zirhuzmorphbpe train korpus/*.txt -o mening_tok --vocab 32000 --min-len 2 [--boundary stem]
```
```python
tok = MorphBPE("mening_tok")
```
- `--boundary all` (standart): har morfema alohida; `stem`: faqat asos|qo'shimchalar chegarasi (qo'shimchalar zanjiri BPE ga).
- `--min-len 2`: bir harfli qo'shimcha qo'shni qo'shimchaga qo'shiladi (`kel+a+di → kel adi`), asosga emas.
- Kirish matni yangi alifboda bo'lsin (ö ğ ş ç); `zirhuzmorph.translit.to_new()` / `from_cyrillic()`.

## Sxema
`▁` = so'z boshi (oldida bo'shliq), keyingi bo'laklar = so'z davomi (SentencePiece uslubi). Model fayli — HF `tokenizers`
formati (`data/tokenizer.json`); kirish matni doim `MorphPre.line()` dan o'tishi kerak — `MorphBPE` sinfi buni o'zi qiladi.

## Natijalar (bir xil 32k lug'at, bir xil 20M GPT, test 2,4 mln harf)
| | token/so'z | token chegarasi morfema chegarasiga to'g'ri | test BPC |
|---|---|---|---|
| oddiy BPE | 1,30 | 26 % | 1,418 |
| MorphBPE-stem | 1,60 | 78 % | 1,414 |
| **MorphBPE-all** | 1,73 | 82 % | **1,402** |

MorphBPE 20–30 % ko'proq token ishlatadi, evaziga morfologik izchillik va bir xil hisoblashda yaxshiroq bit/harf beradi.

O'rgatilgan model: 200 MB tasodifiy namuna (Vikipediya, yangiliklar, badiiy adabiyot — 1,27 GB toza o'zbek korpusidan).

## Litsenziya
MIT.
