Metadata-Version: 2.4
Name: kyrgyz-morph-tagger
Version: 1.0.0
Summary: Morphological tagger for Kyrgyz: UPOS, XPOS, lemma and the full UD feature bundle
Project-URL: Homepage, https://huggingface.co/NurtilekAlibekov/kyrgyz-morph-tagger
Project-URL: Corpus, https://github.com/AlibekovNurtilek/kyrgyz-ud-treebank
Project-URL: Research, https://github.com/AlibekovNurtilek/kyrgyz-morph-tagger
Author-email: Nurtilek Alibekov <nurtilek007007@gmail.com>
License-Expression: PolyForm-Noncommercial-1.0.0
License-File: LICENSE
Keywords: conllu,kyrgyz,lemmatization,low-resource,morphological-tagging,morphology,nlp,pos-tagging,turkic,universal-dependencies
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Requires-Dist: huggingface-hub>=0.23
Requires-Dist: numpy>=1.24
Requires-Dist: torch>=2.2
Requires-Dist: transformers>=4.40
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Description-Content-Type: text/markdown

# kyrgyz-morph-tagger

[![PyPI](https://img.shields.io/pypi/v/kyrgyz-morph-tagger)](https://pypi.org/project/kyrgyz-morph-tagger/)
[![License: PolyForm NC](https://img.shields.io/badge/license-PolyForm%20Noncommercial%201.0.0-orange.svg)](LICENSE)

Morphological tagger for **Kyrgyz**: predicts UPOS, a Kyrgyz-specific XPOS, the lemma, and the
full Universal Dependencies feature bundle for every token.

Kyrgyz is agglutinative — one word stacks number, possession, case, voice, negation, tense, mood
and person onto a single stem, and a tag is only right if *every* feature in the bundle is right.
This package is a PyTorch re-implementation of the [UDPipe 2](https://ufal.mff.cuni.cz/udpipe/2)
architecture, trained on a manually annotated Kyrgyz UD treebank of 8,485 sentences (~98k tokens).

```bash
pip install kyrgyz-morph-tagger
```

```python
from kyrgyz_morph_tagger import Tagger

tagger = Tagger.load()          # downloads the checkpoint once (~48 MB), then caches it

for sentence in tagger.tag("Мектептерибизден эрте келдик."):
    for token in sentence:
        print(token.form, token.lemma, token.upos, token.feats_string)
```

```
Мектептерибизден  Мектептер  NOUN   Case=Abl|Number=Plur|Poss=Yes
эрте              эрте       ADV    _
келдик            кел        VERB   Mood=Ind|Number=Plur|Person=1|Polarity=Pos|Tense=Past|Voice=Act
.                 .          PUNCT  _
```

The first `Tagger.load()` also pulls `xlm-roberta-base` (~1.1 GB) through `transformers` — the
tagger reads frozen contextual embeddings from it. Both downloads are cached, so only the first
run pays. A GPU is optional: the full 425-sentence test set tags in **12 s on CPU**, 2 s on GPU,
with identical output.

## Usage

| Input | Call |
|---|---|
| Raw text (splits sentences, tokenizes) | `tagger.tag(text)` |
| One raw sentence | `tagger.tag_sentence(text)` |
| Already-tokenized sentences | `tagger.tag_pretokenized([["Мектеп", "жакшы", "."]])` |
| A CoNLL-U file (keeps its tokenization and syntax) | `tagger.tag_conllu("in.conllu", "out.conllu")` |

The tagger predicts **morphology only** — `HEAD` and `DEPREL` are left untouched.

```bash
ky-tag "Мектептерибизден келдик."             # print a table
ky-tag -f text.txt --format conllu            # raw text -> CoNLL-U
ky-tag --conllu in.conllu -o out.conllu       # fill an existing CoNLL-U file
```

```python
Tagger.load()                        # released model, from the Hugging Face Hub
Tagger.load("path/to/checkpoint")    # a local directory you trained yourself
Tagger.load(revision="v1.0")         # pin a Hub revision for reproducibility
Tagger.load(device="cpu")
```

## Accuracy

Held-out test set (425 sentences), exact match, **gold tokenization**:

| | UPOS | XPOS | UFeats | AllTags | Lemmas |
|---|--:|--:|--:|--:|--:|
| Released checkpoint (seed 42) | 93.90 | 92.63 | 81.25 | 79.44 | 87.96 |
| Mean ± std over 4 seeds | 93.80 ±0.12 | 92.53 ±0.13 | 80.96 ±0.33 | 79.19 ±0.40 | 88.06 ±0.14 |

`UFeats` is exact-set match of the whole feature bundle; `AllTags` additionally requires UPOS and
XPOS to be right. A single wrong feature invalidates the token — which is why UFeats (81) sits far
below UPOS (94). That is the honest measure of full morphological analysis, and under the same
metric three frontier LLMs given the same tagging scheme in the prompt reach only UFeats 58–60
zero-shot.

**Tokenizer.** `tag()` on raw text must tokenize first, which costs a little accuracy on top of
the table above. The bundled rule-based tokenizer, scored against the corpus's gold tokenization
on the same test set: **token F1 99.27**, **95.3 %** of sentences tokenized exactly right. The
sentence splitter recovers ~85 % of boundaries in running text; the ceiling is low for a real
reason — 11 % of the corpus's sentences end with no final punctuation at all, and those
boundaries cannot be recovered from the surface. If you already have tokens, use
`tag_pretokenized()` and take the tokenizer out of the loop.

## The tagset

Universal POS and features stay standard UD; Kyrgyz-specific detail lives in `XPOS` and in a few
declared feature values.

- **Six cases** — Nom, Gen, Dat, Acc, Loc, Abl. The instrumental `-менен` is analysed as a
  postposition and the similative `-дай` as derivation, so neither is a case.
- **Deverbal categories** — the participle (*atooch*) and the verbal noun (*ktooch*) are
  `UPOS=VERB` with the appropriate `VerbForm`, keeping their fine label in XPOS.
- **Ideophones and interjections** — `UPOS=INTJ`, subtype in XPOS.
- `AdjType=Rel|Qual`, `AdvType=Tim|Loc|Man|Deg`, `VerbForm=Part|Conv|Vnoun`.

17 UPOS tags, 24 XPOS tags, 14 feature categories. The corpus passes the official UD validator
(level 2) with zero errors.

## Limitations

- Morphology only; no syntax.
- Trained on news, web and literary text — other domains will be weaker.
- Sentences longer than 512 XLM-R subwords lose contextual embeddings for the overflow. The
  package warns when this happens; split long sentences.

## Development

```bash
pip install -e ".[dev]"
pytest -m "not model"     # fast: tokenizer and API surface
pytest                    # everything, incl. downloading and running the checkpoint
```

To run the tokenizer's full gold-corpus regression check, point `KY_TREEBANK` at a checkout of
the treebank.

## Licence

**[PolyForm Noncommercial 1.0.0](LICENSE)** — free for research, teaching, personal projects and
noncommercial organisations. **Commercial use requires a separate licence**; contact the licensor.

Be aware of what this means before you depend on it: PolyForm Noncommercial is *not* an
OSI-approved open-source licence, and corporate licence scanners will treat it accordingly. If
you need the tagger inside a product, ask — do not assume.

The **training corpus is CC BY-SA 4.0** and is distributed separately; it is not shipped in this
package, and this licence does not extend to it.

## Related

- **Model weights:** <https://huggingface.co/NurtilekAlibekov/kyrgyz-morph-tagger>
- **Corpus** (CC BY-SA 4.0): <https://github.com/AlibekovNurtilek/kyrgyz-ud-treebank>
- **Research code** — training, experiments, the LLM comparison:
  <https://github.com/AlibekovNurtilek/kyrgyz-morph-tagger>
