Metadata-Version: 2.4
Name: turkicnlp
Version: 0.5.0
Summary: NLP toolkit for Turkic languages
Author-email: Sherzod Hakimov <sherzodhakimov@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/turkic-nlp/turkicnlp
Project-URL: Documentation, https://github.com/turkic-nlp/turkicnlp#readme
Project-URL: Repository, https://github.com/turkic-nlp/turkicnlp
Project-URL: Issues, https://github.com/turkic-nlp/turkicnlp/issues
Keywords: nlp,toolkit,turkic,kazakh,turkish,turkmen,uzbek,azerbaijani,kyrgyz,tatar,uyghur,stanza,linguistics,morphology,pos-tagging,dependency-parsing,asr,ner,machine-translation,embeddings
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Natural Language :: Turkish
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: regex>=2022.1.18
Provides-Extra: hfst
Requires-Dist: hfst>=3.12.0.2b0; extra == "hfst"
Provides-Extra: stanza
Requires-Dist: stanza<1.12,>=1.8; extra == "stanza"
Provides-Extra: lid
Requires-Dist: fasttext-wheel>=0.9; extra == "lid"
Requires-Dist: huggingface_hub>=0.20; extra == "lid"
Provides-Extra: translation
Requires-Dist: torch>=2.0; extra == "translation"
Requires-Dist: transformers>=4.30; extra == "translation"
Requires-Dist: tokenizers>=0.13; extra == "translation"
Requires-Dist: sentencepiece>=0.1.99; extra == "translation"
Provides-Extra: asr
Requires-Dist: omnilingual-asr>=0.2; (python_version >= "3.10" and python_version < "3.13") and extra == "asr"
Requires-Dist: soundfile>=0.12; extra == "asr"
Requires-Dist: torchaudio<2.9,>=2.8; (python_version >= "3.10" and python_version < "3.13") and extra == "asr"
Provides-Extra: transformers
Requires-Dist: torch>=2.0; extra == "transformers"
Requires-Dist: transformers>=4.30; extra == "transformers"
Requires-Dist: tokenizers>=0.13; extra == "transformers"
Requires-Dist: sentencepiece>=0.1.99; extra == "transformers"
Provides-Extra: all
Requires-Dist: hfst>=3.12.0.2b0; extra == "all"
Requires-Dist: stanza<1.12,>=1.8; extra == "all"
Requires-Dist: fasttext-wheel>=0.9; extra == "all"
Requires-Dist: huggingface_hub>=0.20; extra == "all"
Requires-Dist: torch>=2.0; extra == "all"
Requires-Dist: transformers>=4.30; extra == "all"
Requires-Dist: tokenizers>=0.13; extra == "all"
Requires-Dist: sentencepiece>=0.1.99; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Dynamic: license-file

<p align="center">
  <img src="https://sherzod-hakimov.github.io/images/cover.png" alt="TurkicNLP — Six Branches of Turkic Language Family" width="200">
</p>

<h1 align="center">TurkicNLP</h1>

<p align="center">
  <strong>NLP toolkit for 20+ Turkic languages</strong> — a pip-installable Python library inspired by <a href="https://stanfordnlp.github.io/stanza/">Stanza</a>, with adaptations for the low-resource, morphologically rich Turkic language family.
</p>

<p align="center">
  Maintained by <a href="https://sherzod-hakimov.github.io/">Sherzod Hakimov</a>
</p>

<p align="center">
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License: Apache-2.0"></a>
  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10%20|%203.11%20|%203.12-blue.svg" alt="Python 3.10 | 3.11 | 3.12"></a>
  <img src="https://img.shields.io/badge/status-alpha-orange.svg" alt="Status: Alpha">
  <img src="https://img.shields.io/badge/languages-24_Turkic-green.svg" alt="24 Turkic Languages">
  <a href="https://github.com/turkic-nlp/turkicnlp/actions/workflows/test-installation.yml"><img src="https://github.com/turkic-nlp/turkicnlp/actions/workflows/test-installation.yml/badge.svg" alt="Package Installation Tests"></a>
</p>

<p align="center">
  <a href="#installation"><strong>Install</strong></a> ·
  <a href="#quick-start"><strong>Quick start</strong></a> ·
  <a href="ROADMAP.md"><strong>Roadmap</strong></a> ·
  <a href="CONTRIBUTING.md"><strong>Contributing</strong></a> ·
  <a href="CHANGELOG.md">Changelog</a> ·
  <a href="https://arxiv.org/pdf/2602.19174">Paper</a> ·
  <a href="https://github.com/turkic-nlp/turkic-nlp-code-samples">Notebooks</a> ·
  <a href="https://huggingface.co/turkicnlp">🤗 Models &amp; data</a> ·
  <a href="https://discord.gg/CeVTbGpmMQ">Discord</a>
</p>

> **Help us cover every Turkic language.** Native speakers, linguists and developers are all welcome — see
> **[CONTRIBUTING.md](CONTRIBUTING.md)** for how to get started, and **[ROADMAP.md](ROADMAP.md)** for planned
> features, known issues and open ideas you can pick up.

## Citation

If you use TurkicNLP in your research, please cite:

```bibtex
@misc{hakimov2026turkicnlpnlptoolkit,
      title={TurkicNLP: An NLP Toolkit for Turkic Languages}, 
      author={Sherzod Hakimov},
      year={2026},
      eprint={2602.19174},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.19174}, 
}
```

## Links

| | |
|---|---|
| Paper (arXiv) | [arxiv.org/abs/2602.19174](https://arxiv.org/abs/2602.19174) |
| Code samples (Jupyter notebooks) | [turkic-nlp/turkic-nlp-code-samples](https://github.com/turkic-nlp/turkic-nlp-code-samples) |
| Datasets & models | [🤗 huggingface.co/turkicnlp](https://huggingface.co/turkicnlp) |
| Community | [Discord](https://discord.gg/CeVTbGpmMQ) |
| Planned work and known issues | [ROADMAP.md](ROADMAP.md) |
| How to contribute | [CONTRIBUTING.md](CONTRIBUTING.md) |

## Features

- **24 Turkic languages** from Turkish to Sakha, Kazakh to Uyghur
- **Script-aware from the ground up** — Latin, Cyrillic, Perso-Arabic, Old Turkic Runic
- **Automatic script detection** and bidirectional transliteration
- **[Apertium FST morphology](https://wiki.apertium.org/wiki/Turkic_languages)** for ~20 Turkic languages via Python-native `hfst` bindings (no system install)
- **Stanza/UD integration** — pretrained tokenization, POS tagging, lemmatization and dependency parsing via [Stanza](https://stanfordnlp.github.io/stanza/) models trained on [Universal Dependencies](https://universaldependencies.org/) treebanks (9 languages, 5 of them custom-trained), plus NER for Turkish and Kazakh
- **Morpheme segmentation (`MorphemeTokenizer`)** — splits words into labelled morphemes by combining Apertium and the neural analyzer (89% exact segmentation on a 483-word test set in 20 languages)
- **Morpheme-aware subword tokenization (`sp_tokenize`)** — SentencePiece over morpheme-segmented text, for training language models
- **NLLB embeddings + translation backend** — sentence/document vectors and MT via [NLLB-200](https://huggingface.co/facebook/nllb-200-distilled-600M) (11 Turkic languages)
- **Language identification (`LanguageDetection`, GlotLID model)** — FastText-based LID with 1,000+ Glottolog language labels
- **Speech recognition (`SpeechRecognizer`)** — speech-to-text for 20 Turkic languages via Meta's [Omnilingual ASR](https://github.com/facebookresearch/omnilingual-asr), with output in TurkicNLP scripts and a speech → text → analysis pipeline
- **Multilingual Glot500 neural models** — POS tagging & dependency parsing (15 languages), morphological analysis & lemmatization (22 TurkicNLP languages) via shared [Glot500](https://github.com/cisnlp/Glot500) backbone
- **LLM backend** — use any LLM for morphology, NER, translation, sentiment and spelling correction in all 24 languages: a local model served with [vLLM](https://docs.vllm.ai) or [Ollama](https://ollama.com), OpenAI, Claude, or an in-process Hugging Face model ([details](#llm-backend-local-vllm--ollama-openai-claude-or-a-hugging-face-model))
- **Multiple backends** — choose between rule-based, Apertium FST, Stanza, Glot500 neural or LLM backends per processor
- **License isolation** — library is Apache-2.0; Apertium GPL-3.0 data downloaded separately
- **Stanza-compatible API** — `Pipeline`, `Document`, `Sentence`, `Word`

## Installation

**Requirements:** Python 3.10 – 3.12.

TurkicNLP has a small core and optional extras for each group of components:

```bash
pip install turkicnlp                    # core — tokenization, rule-based processing, transliteration, CoNLL-U I/O
pip install "turkicnlp[hfst]"           # + Apertium FST morphology (Linux and macOS only)
pip install "turkicnlp[stanza]"         # + Stanza neural models (tokenize, POS, lemma, depparse, NER)
pip install "turkicnlp[lid]"            # + Language detection (GlotLID model; FastText + HF weights)
pip install "turkicnlp[translation]"    # + NLLB embeddings and machine translation
pip install "turkicnlp[transformers]"   # + Glot500 multilingual POS/DepParse/Morph models
pip install "turkicnlp[all]"            # all text components above (Linux and macOS only)
pip install "turkicnlp[asr]"            # + speech recognition (Omnilingual ASR) — see below, use a separate environment
pip install "turkicnlp[dev]"            # development tools (pytest, black, ruff, mypy)
```

### Choosing an environment: text-only vs. speech + text

Speech recognition is built on [`omnilingual-asr`](https://github.com/facebookresearch/omnilingual-asr), whose
dependency `fairseq2` pins `torch==2.8`, `numpy<2` and `huggingface_hub<1`, and therefore `transformers` 4.x.
All other TurkicNLP components also work with these versions, so there are two tested setups:

| | Text environment | Speech + text environment |
|---|---|---|
| Extras | `[all]` | `[all,asr]` |
| Use it for | all text components | speech recognition **and** all text components |
| Python | 3.10 – 3.12 | 3.10 – 3.12 |
| torch / torchaudio | 2.10 / – | 2.8 / 2.8 |
| numpy | 2.x | 1.26 |
| transformers / huggingface_hub | 5.x / 1.x | 4.57 / 0.36 |
| stanza | 1.11 | 1.11 |
| Exact versions | [`requirements/lock-text.txt`](requirements/lock-text.txt) | [`requirements/lock-speech.txt`](requirements/lock-speech.txt) |

> ⚠️ Do not add `[asr]` to an existing text environment: it downgrades torch, numpy and huggingface_hub,
> which breaks `transformers` 5.x. Create a fresh environment instead.

#### Text environment

```bash
python3 -m venv venv
source venv/bin/activate              # Windows: venv\Scripts\activate
pip install --upgrade pip
pip install "turkicnlp[all]"          # from PyPI
# or, from a clone of this repository:
pip install -e ".[all]"
```

#### Speech + text environment

System requirements: `libsndfile` and a C++ compiler with CMake (`kenlm`, a dependency of `omnilingual-asr`, is compiled from source).

```bash
# macOS
brew install libsndfile cmake
# Ubuntu / Debian
sudo apt-get install libsndfile1 cmake build-essential

python3.10 -m venv venv-asr
source venv-asr/bin/activate
pip install --upgrade pip
pip install "turkicnlp[all,asr]"      # from PyPI
# or, from a clone of this repository:
pip install -e ".[all,asr]"
```

#### Reproducing the tested versions

The lock files pin every package of the two tested environments (Python 3.10, macOS arm64):

```bash
pip install -r requirements/lock-text.txt   && pip install -e .   # text environment
pip install -r requirements/lock-speech.txt && pip install -e .   # speech + text environment
```

#### Version constraints and known issues

- **Stanza < 1.12.** The custom Azerbaijani, Uzbek, Turkmen, Tatar and Bashkir Stanza models were trained with Stanza 1.11;
  Stanza 1.15 fails to load their parsers (`size mismatch for deprel.scorer`). The extras enforce `stanza>=1.8,<1.12`.
- **sentencepiece** is needed to build the Glot500 (XLM-R) tokenizer with `transformers` 4.x; it is included in the
  `[transformers]`, `[translation]` and `[all]` extras.
- **torchaudio 2.8.** `omnilingual-asr` does not pin torchaudio; newer versions fail to load against torch 2.8
  (`Symbol not found: _torch_library_impl`). The `[asr]` extra pins `torchaudio>=2.8,<2.9`.
- **`kenlm` build fails on macOS with `'climits' file not found`.** A stale
  `/Library/Developer/CommandLineTools/usr/include/c++/v1` left by an old Command Line Tools install shadows the SDK's C++ headers.
  Either remove it once:
  ```bash
  sudo mv /Library/Developer/CommandLineTools/usr/include/c++ /Library/Developer/CommandLineTools/usr/include/c++.old
  ```
  or point the compiler at the SDK headers for the install:
  ```bash
  export CXXFLAGS="-nostdinc++ -isystem $(xcrun --show-sdk-path)/usr/include/c++/v1"
  export CMAKE_POLICY_VERSION_MINIMUM=3.5      # kenlm's CMakeLists predates CMake 4
  pip install -e ".[all,asr]"
  ```
- **Memory for speech recognition.** `omniASR_LLM_1B` (9 GB checkpoint) runs in about 6 GB of RAM with the default
  low-memory loading (memory-mapped checkpoint, bfloat16 weights). For smaller machines use `omniASR_LLM_300M`
  or a CTC model (`omniASR_CTC_300M`, 1.3 GB).

Open issues of the components themselves (e.g. low Apertium coverage for some languages) are tracked in
[ROADMAP.md → Known issues](ROADMAP.md#2-known-issues).

### Verify the installation

```python
import turkicnlp

# Core (no downloads)
from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator
print(Transliterator("kaz", Script.CYRILLIC, Script.LATIN).transliterate("Қазақстан"))   # Qazaqstan

# Text components (downloads models on first use)
nlp = turkicnlp.Pipeline("kaz", processors=["tokenize", "pos", "lemma", "depparse"])
doc = nlp("Мен мектепке бардым.")
print([(w.text, w.upos, w.lemma) for w in doc.words])
# [('Мен', 'PRON', 'мен'), ('мектепке', 'NOUN', 'мектеп'), ('бардым', 'VERB', 'бар'), ('.', 'PUNCT', '.')]

# Speech environment only
print(turkicnlp.list_asr_languages(check_model=True))   # {'tur': {'Latn': 'tur_Latn'}, 'aze': {...}, ...}
```

### Platform compatibility

Installation tests run on every push and pull request and nightly, across all combinations of OS, Python version and install extra below; the test suite (`pytest -m "not slow"`) runs on Ubuntu and macOS (see [CI workflow](https://github.com/turkic-nlp/turkicnlp/actions/workflows/test-installation.yml)).

| Extra | Ubuntu 22.04 / 24.04 | macOS 14 / 15 | Windows 2025 |
|---|---|---|---|
| base | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 |
| `[hfst]` | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 | ❌ not available |
| `[stanza]` | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 |
| `[lid]` | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 |
| `[transformers]` | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 |
| `[translation]` | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 |
| `[all]` | ✅ 3.9 – 3.12 | ✅ 3.9 – 3.12 | ❌ not available |
| `[asr]` | not yet in CI | ✅ 3.10 (manually tested) | not tested |

> **Windows users:** the `hfst` Python package has no published wheels for Python 3.7 or later on Windows — this is an upstream limitation with no current workaround. All features except Apertium FST morphology work normally on Windows; use `turkicnlp[stanza]` or `turkicnlp[translation]` instead. If you need Apertium FST morphology on Windows, the recommended approach is [Windows Subsystem for Linux (WSL)](https://wiki.apertium.org/wiki/Apertium_on_Windows), where `hfst` installs normally.

## Quick Start

```python
import turkicnlp

# Download models for a language
turkicnlp.download("kaz")

# Build a pipeline
nlp = turkicnlp.Pipeline("kaz", processors=["tokenize", "pos", "lemma", "ner", "depparse"])

# Process text
doc = nlp("Мен мектепке бардым")

# Access annotations
for sentence in doc.sentences:
    for word in sentence.words:
        print(f"{word.text}\t{word.lemma}\t{word.upos}\t{word.feats}")

# Export to CoNLL-U
print(doc.to_conllu())
```

`turkicnlp.Pipeline("kaz")` without `processors=` loads the default set for the language: the tokenizer and the
morphology, POS, lemma, feature, parsing and NER processors listed in the catalog (Apertium morphology is preferred
over the neural analyzer when both exist). Processors that download large models or need an extra environment —
`translate`, `embeddings`, `sp_tokenize`, `asr` — are loaded only when you name them. See
[Processors and backends](#processors-and-backends) for all names.

### Embeddings (NLLB)

```python
import math
import turkicnlp

turkicnlp.download("tur", processors=["embeddings"])
nlp = turkicnlp.Pipeline("tur", processors=["embeddings"])

doc1 = nlp("Bugün hava çok güzel ve parkta yürüyüş yaptım.")
doc2 = nlp("Parkta yürüyüş yapmak bugün çok keyifliydi çünkü hava güzeldi.")

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    return dot / (norm_a * norm_b)

print(len(doc1.embedding), len(doc2.embedding))
print(f"cosine = {cosine_similarity(doc1.embedding, doc2.embedding):.4f}")
print(doc1._processor_log)  # ['embeddings:nllb']
```

### Language ID (`LanguageDetection`, GlotLID model)

```python
import turkicnlp

lid = turkicnlp.LanguageDetection()  # defaults to all Turkic languages supported by TurkicNLP
labels, probs = lid.predict("salam, hemmelere!", k=3)
print(labels, probs)

# Limit to specific labels
limited = turkicnlp.LanguageDetection(
    languages=[
        "__label__eng_Latn",
        "__label__tur_Latn",
        "__label__kaz_Cyrl",
    ]
)
print(limited.predict("Merhaba dünya!", k=1))
```

### Speech Recognition (Omnilingual ASR)

`SpeechRecognizer` wraps Meta's [Omnilingual ASR](https://github.com/facebookresearch/omnilingual-asr) models
([`omniASR_LLM_1B`](https://huggingface.co/facebook/omniASR-LLM-1B) by default) behind TurkicNLP conventions:

- languages are given as TurkicNLP ISO 639-3 codes (`kaz`, `uzb`, …) and mapped to Omnilingual codes (`kaz_Cyrl`, `uzn_Latn`, …);
- only the 20 Turkic languages below are accepted; each code is checked against the model's `supported_langs`,
  and non-Turkic languages (`eng`, `rus`, …) are rejected;
- transcripts are returned in the language's primary TurkicNLP script (or the one you request), using the toolkit's
  transliterator where the model writes another script;
- recordings longer than the model's 40-second limit are split at pauses and the pieces are transcribed and joined;
- the model is loaded once and shared by all languages; by default it is loaded memory-mapped in bfloat16 (≈ 6 GB RAM).

Requires the [speech + text environment](#speech--text-environment). The model weights are downloaded on first use
(9.1 GB for `omniASR_LLM_1B`) and cached in `~/.cache/fairseq2/assets/`.

#### Transcribe audio files

```python
import turkicnlp

asr = turkicnlp.SpeechRecognizer(
    model_card="omniASR_LLM_1B",   # default; see the model table below
    device="cpu",                  # default: "cuda" if available, else "cpu"
    dtype="bfloat16",              # default ("auto"): bfloat16, or float16 on Apple "mps"
    low_memory=True,               # default: memory-map the checkpoint while loading
)

files = ["asr_samples/kaz_sample.wav", "asr_samples/tur_sample.wav"]
texts = asr.transcribe(files, lang=["kaz", "tur"], batch_size=2)
print(texts)
# ['мен бүгін мектепке бардым алматы қазақстанның ең үлкен қаласы',
#  "bugün hava çok güzel ali dün ankara'ya gitti"]
```

The interface mirrors Omnilingual ASR's `ASRInferencePipeline.transcribe(audio_files, lang=..., batch_size=...)`;
`lang` is one code for all inputs or a list with one code per input. A single input returns a string, a list returns a list.

#### Segments, details and output script

```python
res = asr.transcribe("asr_samples/kaz_sample.wav", lang="kaz", return_details=True)
print(res.lang, res.model_lang, res.script)   # kaz kaz_Cyrl Cyrl
print(res.text)
for seg in res.segments:                      # time-aligned pieces (long audio is split at pauses)
    print(f"[{seg.start:.2f}-{seg.end:.2f}] {seg.text}")

# Ask for another script: Kazakh in the 2021 Latin alphabet
print(asr.transcribe("asr_samples/kaz_sample.wav", lang="kaz", script="Latn"))
# men bügın mektepke bardym almaty qazaqstannyñ eñ ülken qalasy

# The model writes Karakalpak and Crimean Tatar in Cyrillic; TurkicNLP returns their primary Latin script
res = asr.transcribe("kaa_audio.wav", lang="kaa", return_details=True)
print(res.model_text)   # Cyrillic, as produced by the model
print(res.text)         # Latin (2016 Karakalpak alphabet)
```

#### Other input types

```python
import soundfile as sf

wav, sr = sf.read("asr_samples/tur_sample.wav")
asr.transcribe(wav, lang="tur", sample_rate=sr)                         # waveform array
asr.transcribe({"waveform": wav, "sample_rate": sr}, lang="tur")        # dict
asr.transcribe(open("asr_samples/tur_sample.wav", "rb").read(), lang="tur")   # encoded bytes

# Hugging Face datasets audio columns work directly
# from datasets import load_dataset
# ds = load_dataset(..., split="test")
# asr.transcribe(list(ds["audio"][:8]), lang="kir", batch_size=4)
```

Stereo audio is converted to mono and resampled to 16 kHz automatically.

#### Speech → text → analysis

```python
nlp = turkicnlp.Pipeline(
    "kaz",
    processors=["asr", "tokenize", "pos", "lemma", "depparse"],
    asr_model_card="omniASR_LLM_1B",   # asr_* options configure the recognizer
    asr_device="cpu",                  # also: asr_dtype, asr_batch_size, asr_max_segment_seconds
)
doc = nlp.from_audio("asr_samples/kaz_sample.wav")
print(doc.text)
print(doc.audio_segments)              # [{'start': 0.0, 'end': 4.86, 'text': '...'}]
for w in doc.words:
    print(w.text, w.upos, w.lemma, w.head, w.deprel)
```

Omnilingual ASR outputs lowercase text without punctuation, so tagging and parsing quality on transcripts can be lower
than on edited text.

#### Command line

```bash
python run_asr.py asr_samples/kaz_sample.wav asr_samples/tur_sample.wav --lang kaz tur --device cpu
python run_asr.py lecture.wav --lang tat --model omniASR_LLM_300M      # smaller model
python run_asr.py interview.wav --lang uzb --script Cyrl --analyze      # + tokenize/POS/depparse
```

`asr_samples/` contains two short synthetic test recordings (Kazakh and Turkish).

#### Models and memory

Any Omnilingual ASR model card can be passed as `model_card`:

| Model card | Checkpoint | Notes |
|---|---|---|
| `omniASR_LLM_300M` | 6.5 GB | smallest LLM model |
| `omniASR_LLM_1B` (default) | 9.1 GB | ≈ 6 GB RAM with low-memory loading (tested on a 16 GB MacBook) |
| `omniASR_LLM_3B` | 17.5 GB | GPU recommended |
| `omniASR_LLM_7B` | 31.2 GB | best quality; GPU required in practice |
| `omniASR_CTC_300M` / `_1B` / `_3B` | 1.3 / 3.9 / 12.3 GB | faster, no language conditioning (the language code is ignored) |

If the process is killed for lack of memory, keep `low_memory=True` and `dtype="bfloat16"` (the defaults) or use a smaller model.
`dtype="float32"` doubles the memory needed.

#### Supported languages

| Language | Code | Omnilingual code(s) | Output script |
|---|---|---|---|
| Turkish | `tur` | `tur_Latn` | Latin |
| Azerbaijani | `aze` | `aze_Latn`, `aze_Cyrl` | Latin |
| South Azerbaijani | `azb` | `aze_Arab` | Perso-Arabic |
| Turkmen | `tuk` | `tuk_Latn` | Latin |
| Gagauz | `gag` | `gag_Latn` | Latin |
| Kazakh | `kaz` | `kaz_Cyrl` | Cyrillic (Latin via transliteration) |
| Kyrgyz | `kir` | `kir_Cyrl` | Cyrillic |
| Tatar | `tat` | `tat_Cyrl` | Cyrillic (Latin via transliteration) |
| Bashkir | `bak` | `bak_Cyrl` | Cyrillic |
| Crimean Tatar | `crh` | `crh_Cyrl` | Latin (transliterated) |
| Karakalpak | `kaa` | `kaa_Cyrl` | Latin (transliterated) |
| Nogai | `nog` | `nog_Cyrl` | Cyrillic |
| Kumyk | `kum` | `kum_Cyrl` | Cyrillic |
| Karachay-Balkar | `krc` | `krc_Cyrl` | Cyrillic |
| Uzbek | `uzb` | `uzn_Latn`, `uzb_Cyrl` | Latin |
| Uyghur | `uig` | `uig_Arab`, `uig_Cyrl` | Perso-Arabic |
| Sakha | `sah` | `sah_Cyrl` | Cyrillic |
| Altai | `alt` | `alt_Cyrl` | Cyrillic |
| Khakas | `kjh` | `kjh_Cyrl` | Cyrillic |
| Chuvash | `chv` | `chv_Cyrl` | Cyrillic |

Not available: Tuvan, Khalaj, Ottoman Turkish and Old Turkic (no Omnilingual language code).
`turkicnlp.list_asr_languages(check_model=True)` lists the codes supported by the installed model.

### Machine Translation (NLLB)

```python
import turkicnlp

# Downloads once into ~/.turkicnlp/models/huggingface/facebook--nllb-200-distilled-600M
turkicnlp.download("tur", processors=["translate"])

nlp = turkicnlp.Pipeline(
    "tur",
    processors=["translate"],
    translate_tgt_lang="eng",
)

doc = nlp("Bugün hava çok güzel ve parkta yürüyüş yaptım.")
print(doc.translation)
print(doc._processor_log)  # ['translate:nllb']
```

HTML entities that NLLB sometimes produces (`&apos;`, `&quot;`) are converted back to plain characters.
`translate_tgt_lang` accepts either ISO-639-3 (`"eng"`, `"tuk"`, `"kaz"`) or explicit [Flores-200 codes](https://github.com/facebookresearch/flores/tree/main/flores200#languages-in-flores-200) (`"eng_Latn"`, `"kaz_Cyrl"`).

### LLM Backend (local vLLM / Ollama, OpenAI, Claude, or a Hugging Face model)

Five processors (six backends) can use a large language model, for **all 24 languages** — including
languages without any trained model, such as Khakas, Tuvan or Nogai. The toolkit sends the request, so you
only say *which* LLM to use: a server URL plus an API key, or a local Hugging Face model id. The OpenAI-style
API is spoken by OpenAI, Azure OpenAI, vLLM, Ollama, llama.cpp, LM Studio, OpenRouter and others; Claude is
supported through Anthropic's OpenAI-compatible endpoint. No extra package is needed for the API case.

| Backend | What the LLM does | Writes |
|---|---|---|
| `morph=llm` | lemma, UPOS and UD features for every token | `word.lemma/upos/feats` |
| `morph=apertium_llm` | Apertium analyses as usual; for ambiguous words the LLM **chooses** one of the transducer's readings (it cannot invent an analysis); a configured `apertium_tagger_cmd` still decides first, and all sentences go to the LLM as one parallel batch | `word.lemma/upos/feats` |
| `ner=llm` | entity spans over the toolkit's tokens (labels: `ner_labels=["PER", "LOC", "ORG"]`) | `word.ner`, `sentence.entities` |
| `translate=llm` | translation into `translate_tgt_lang` (ISO code or language name) | `sentence.translation`, `doc.translation` |
| `sentiment=llm` | `positive` / `negative` / `neutral` (or `sentiment_labels=[...]`) | `sentence.sentiment` |
| `spellcheck=llm` | spelling correction; answers that rewrite >30% of a sentence are rejected | `sentence.correction`, `sentence.corrections`, `doc.corrected_text` |

#### 1. Local model with vLLM (GPU server)

Start an OpenAI-compatible server with a small model (Linux + NVIDIA GPU; install vLLM in its own
environment, `pip install vllm`):

```bash
bash scripts/start_vllm_server.sh                                   # Qwen/Qwen2.5-1.5B-Instruct on :8000
MODEL=Qwen/Qwen3-8B PORT=8001 bash scripts/start_vllm_server.sh     # another model / port
GPUS=0,1 TP=2 MODEL=Qwen/Qwen3-32B bash scripts/start_vllm_server.sh # two GPUs
```

Then use it from TurkicNLP (from another machine, replace `localhost` with the server name):

```python
import turkicnlp

nlp = turkicnlp.Pipeline(
    "kaz",
    processors=["tokenize", "morph", "ner", "sentiment", "spellcheck", "translate"],
    morph_backend="llm", ner_backend="llm", sentiment_backend="llm", translate_backend="llm",
    translate_tgt_lang="eng",
    llm_model="Qwen/Qwen2.5-1.5B-Instruct",       # the model the server was started with
    llm_base_url="http://localhost:8000/v1",
    llm_api_key="turkicnlp-local",                # API_KEY of start_vllm_server.sh
)
doc = nlp("Қасым-Жомарт Тоқаев кеше Астанада шетелдік инвесторлармен кездесті.")

for w in doc.words:
    print(w.text, w.lemma, w.upos, w.feats)
print(doc.entities, doc.sentences[0].sentiment, doc.corrected_text, doc.translation)
```

Without a GPU (e.g. on a Mac) use [Ollama](https://ollama.com) the same way:
`ollama pull qwen2.5:7b`, then `llm_model="qwen2.5:7b", llm_base_url="http://localhost:11434/v1", llm_api_key="ollama"`.

#### 2. OpenAI

```python
import os, turkicnlp

os.environ["OPENAI_API_KEY"] = "sk-..."           # or pass llm_api_key="sk-..."

nlp = turkicnlp.Pipeline(
    "uzb",
    processors=["tokenize", "ner", "translate"],
    ner_backend="llm", translate_backend="llm", translate_tgt_lang="deu",
    llm_model="gpt-4.1-mini",                     # any chat model of your account
)                                                  # llm_base_url defaults to https://api.openai.com/v1
doc = nlp("Shavkat Mirziyoyev Toshkentda yangi maktabni ochdi.")
print(doc.entities, doc.translation)
```

#### 3. Claude (Anthropic)

```python
import os, turkicnlp

os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."    # or pass llm_api_key="sk-ant-..."

nlp = turkicnlp.Pipeline(
    "tyv",
    processors=["tokenize", "morph", "spellcheck"],
    morph_backend="llm",
    llm_provider="anthropic",                     # https://api.anthropic.com/v1, key from ANTHROPIC_API_KEY
    llm_model="claude-sonnet-4-6",
)
doc = nlp("Мен бөгүн школага бардым.")
print([(w.text, w.lemma, w.upos, w.feats) for w in doc.words], doc.corrected_text)
```

Model names change; use any model your account offers. With `llm_provider="anthropic"` no `temperature`
is sent, because recent Claude models only accept the default.

#### 4. Hugging Face model in-process (no server)

```python
nlp = turkicnlp.Pipeline("tur", processors=["tokenize", "ner"], ner_backend="llm",
                         llm_provider="hf", llm_model="Qwen/Qwen2.5-1.5B-Instruct")
```

Needs `pip install "turkicnlp[transformers]"`; requests run one at a time, so this is for small models and
small amounts of text. For anything larger, serve the model with vLLM.

#### Example run

`examples/llm_backend_demo.py` runs all LLM backends on one text (`--backend vllm | ollama | openai | claude | hf`):

```text
$ python examples/llm_backend_demo.py --backend hf --model Qwen/Qwen2.5-1.5B-Instruct --lang kaz

Model: Qwen/Qwen2.5-1.5B-Instruct (hf)
Text:  Қасым-Жомарт Тоқаев кеше Астанада шетелдік инвесторлармен кездесті. Бұл кездесу өте пайдалы болды.

Morphology:
  Қасым-Жомарт
  Тоқаев
  кеше
  Астанада
  шетелдік
  инвесторлармен
  кездесті
  .
  Бұл                    бұл                DET    Case=Gen|Definite=Def|Num=One
  кездесу                кездесу            NOUN   Case=Acc|Gender=Fem|Number=Sing|PronType=Dem
  өте                    өте                ADP    Case=Loc
  пайдалы                пайда              NOUN   Case=Acc|Gender=Masc|Number=Sing|PronType=Dem
  болды                  болды              VERB   Aspect=Perf|Mood=Ind|Number=Sing|Person=3|Polarity=Pos|Tense=Pres|VerbForm=Fin
  .                      .                  PUNCT  _

Entities:    [('Қасым-Жомарт Тоқаев', 'PER'), ('Астанада', 'LOC'), ('инвесторлармен', 'ORG')]
Sentiment:   ['positive', 'positive']
Spelling:    Қазым-Жомарт Тоқаев кеше Астанада шетелдік инвесторлармен кездесті. Бұл кездесу өте пайдалы болды. [('Қасым-Жомарт', 'Қазым-Жомарт')]
Translation: Kasym-Jomart Tokayev metalled in Astana to meet investors. It is very useful.

Provenance:  ['tokenize:stanza', 'spellcheck:llm(Qwen/Qwen2.5-1.5B-Instruct, spellcheck/v1)',
              'morph:llm(Qwen/Qwen2.5-1.5B-Instruct, morph/v2)[1 failed]', 'ner:llm(Qwen/Qwen2.5-1.5B-Instruct, ner/v1)',
              'sentiment:llm(Qwen/Qwen2.5-1.5B-Instruct, sentiment/v1)', 'translate:llm(Qwen/Qwen2.5-1.5B-Instruct, translate/v1)']
Time: 274.7s
```

This run used a 1.5B-parameter model on a laptop, which shows both how the backend works and why the model
matters:

* **Validated output.** For the first sentence the model's morphology answer did not cover the tokens, even
  after a retry with feedback, so those fields stay empty and the log says `[1 failed]` — the toolkit does not
  write shifted or invented analyses. With `morph_on_error="raise"` it would stop instead.
* **Spelling guard.** Word changes that are not spelling fixes (rewritten suffixes, added words) are reverted;
  a plausible-looking one-letter change such as *Қасым → Қазым* still gets through, which is a model error.
* **Quality.** Lemmas, features, the *ORG* label and "metalled" are wrong: a 1.5B model is enough to test the
  set-up, not for Kazakh analysis. Use a larger model (e.g. Qwen3-32B on vLLM, GPT or Claude) for real work,
  and evaluate it on your language first.

#### How it works and where the prompt is

Each LLM backend is an ordinary pipeline processor. It fills its prompt template — one versioned file per
task in `turkicnlp/resources/prompts/*.json` — with the language, script, the toolkit's tokens and a few
examples, asks the model for JSON, validates the answer (and asks once more if it is unusable) and writes
the normal `Document` fields. `doc._processor_log` records model and prompt version, e.g.
`ner:llm(Qwen/Qwen2.5-1.5B-Instruct, ner/v1)`. Replace a prompt with `ner_prompt_file="my_ner.json"` (in a template only `{name}` placeholders of
known variables are filled; other braces, e.g. a JSON example, are left as written) or
add examples with `ner_examples=[{"vars": {"tokens": [...]}, "answer": {...}}]`.

**Settings** (`llm_*` applies to all LLM processors; `<processor>_llm_*` overrides one of them, e.g.
`ner_llm_model="gpt-4.1"`):

| Option | Default | |
|---|---|---|
| `llm_model` | env `TURKICNLP_LLM_MODEL` | model name / Hugging Face id |
| `llm_base_url` | env `TURKICNLP_LLM_BASE_URL`, else `https://api.openai.com/v1` | e.g. `http://localhost:8000/v1` (vLLM), `http://localhost:11434/v1` (Ollama) |
| `llm_api_key` | env `TURKICNLP_LLM_API_KEY`; `OPENAI_API_KEY` only when `llm_base_url` is `api.openai.com`; `ANTHROPIC_API_KEY` for `anthropic` | sent as `Authorization: Bearer …`; masked in `repr(LLMConfig)` (as are header values) |
| `llm_provider` | `openai` | `anthropic` (Claude), `hf` (local `transformers` model) |
| `llm_headers`, `llm_extra_body` | – | extra HTTP headers (e.g. Azure `api-key`) / request fields |
| `llm_temperature`, `llm_max_tokens` | 0.0, 2048 | `llm_temperature=None` sends no temperature |
| `llm_timeout`, `llm_max_retries`, `llm_max_workers` | 120 s, 3, 4 | parallel requests per document; `Retry-After` waits are capped at 60 s |
| `llm_json_mode` | `auto` | JSON schema → JSON object → plain prompt, stepping down only when the server answers that the `response_format` is unsupported |
| `llm_cache`, `llm_cache_dir` | on, `~/.turkicnlp/llm_cache.sqlite` | identical requests are answered from the cache; only answers that passed validation are stored; an unwritable cache location disables the cache with a warning |
| `<processor>_on_error` | `warn` | `raise` to stop on unusable answers (an unusable answer — invalid JSON, an HTML error page, a dropped connection — only affects its own sentence) |
| `translate_tgt_lang`, `translate_tgt_script` | `eng` | target as code (`uzb`), code with script (`uzb_Cyrl`) or name; the prompt names the script, e.g. "Uzbek (Cyrillic script)" |
| `llm_client` | – | pass one `turkicnlp.LLMClient` object to share it |

Free-form questions with the toolkit's analyses as context:

```python
doc = turkicnlp.Pipeline("kaz", processors=["tokenize", "morph"])("Мен кеше мектепке бардым.")
print(turkicnlp.llm.ask("Explain the verb form in this sentence.", doc=doc,
                        llm_provider="anthropic", llm_model="claude-sonnet-4-6"))
```

The context sent with the question is limited to `max_words=400` words (`ask(..., max_words=...)`).

> LLM quality differs strongly between languages (good for Turkish, Kazakh, Uzbek, Azerbaijani; often weak
> for Khakas, Tuvan, Nogai or Kumyk). Evaluate a model for your language before relying on it, and remember
> that the text is sent to the server you configure.

### Using the Stanza Backend

```python
from turkicnlp.processors.stanza_backend import (
    StanzaTokenizer, StanzaPOSTagger, StanzaLemmatizer, StanzaNERProcessor, StanzaDepParser
)
from turkicnlp.models.document import Document

# Models are downloaded automatically on first use
doc = Document(text="Merhaba dünya.", lang="tur")

for Proc in [StanzaTokenizer, StanzaPOSTagger, StanzaLemmatizer, StanzaNERProcessor, StanzaDepParser]:
    proc = Proc(lang="tur")
    proc.load()
    doc = proc.process(doc)

for word in doc.words:
    print(f"{word.text:12} {word.upos:6} {word.lemma:12} head={word.head} {word.deprel}")

# Export to CoNLL-U
print(doc.to_conllu())
```

### Mixed Backends

```python
from turkicnlp.processors.tokenizer import RegexTokenizer
from turkicnlp.processors.stanza_backend import StanzaPOSTagger, StanzaNERProcessor, StanzaDepParser
from turkicnlp.models.document import Document

doc = Document(text="Мен мектепке бардым.", lang="kaz")

# Rule-based tokenizer + Stanza POS/parsing (pretokenized mode)
tokenizer = RegexTokenizer(lang="kaz")
tokenizer.load()
doc = tokenizer.process(doc)

pos = StanzaPOSTagger(lang="kaz")
pos.load()
doc = pos.process(doc)

ner = StanzaNERProcessor(lang="kaz")
ner.load()
doc = ner.process(doc)

parser = StanzaDepParser(lang="kaz")
parser.load()
doc = parser.process(doc)
```

### Multi-Script Support

```python
# Kazakh — auto-detects Cyrillic vs Latin
doc = nlp("Мен мектепке бардым")    # Cyrillic
doc = nlp("Men mektepke bardym")     # Latin

# Explicit script selection
nlp_cyrl = turkicnlp.Pipeline("kaz", script="Cyrl")
nlp_latn = turkicnlp.Pipeline("kaz", script="Latn")

# Transliteration bridge — run Cyrillic model on Latin input
nlp = turkicnlp.Pipeline("kaz", script="Latn", transliterate_to="Cyrl")
```

### Uyghur (Perso-Arabic)

```python
nlp_ug = turkicnlp.Pipeline("uig", script="Arab")
doc = nlp_ug("مەن مەكتەپكە باردىم")
```

### Transliteration

The `Transliterator` class converts text between scripts for any supported language pair:

```python
from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator

# Kazakh Cyrillic → Latin (2021 official alphabet)
t = Transliterator("kaz", Script.CYRILLIC, Script.LATIN)
print(t.transliterate("Қазақстан Республикасы"))
# → Qazaqstan Respublikasy

# Uzbek Latin → Cyrillic
t = Transliterator("uzb", Script.LATIN, Script.CYRILLIC)
print(t.transliterate("O'zbekiston Respublikasi"))
# → Ўзбекистон Республикаси

# Uyghur Perso-Arabic → Latin (ULY)
t = Transliterator("uig", Script.PERSO_ARABIC, Script.LATIN)
print(t.transliterate("مەكتەپ"))
# → mektep

# Azerbaijani Latin → Cyrillic
t = Transliterator("aze", Script.LATIN, Script.CYRILLIC)
print(t.transliterate("Azərbaycan"))
# → Азәрбайҹан

# Turkmen Latin → Cyrillic
t = Transliterator("tuk", Script.LATIN, Script.CYRILLIC)
print(t.transliterate("Türkmenistan"))
# → Түркменистан

# Tatar Cyrillic → Latin (Zamanälif)
t = Transliterator("tat", Script.CYRILLIC, Script.LATIN)
print(t.transliterate("Татарстан Республикасы"))
# → Tatarstan Respublikası
```

#### Common Turkic Script (CTS)

Every supported language except Old Turkic (23 languages) can be converted to the [Common Turkic Alphabet](https://en.wikipedia.org/wiki/Common_Turkic_alphabet) — a unified (34 letters) Latin-based script designed for cross-language interoperability across the Turkic family. Turkish, which already uses a CTS-compatible alphabet, converts with no changes to most letters:

```python
from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator

# Turkish Latin → CTS (near-identical alphabets)
t = Transliterator("tur", Script.LATIN, Script.COMMON_TURKIC)
print(t.transliterate("Türkçe güzel bir dil"))
# → Türkçe güzel bir dil

# Other languages gain a shared representation
t = Transliterator("kaz", Script.CYRILLIC, Script.COMMON_TURKIC)
print(t.transliterate("Қазақстан"))
# → Qazaqstan

t = Transliterator("aze", Script.LATIN, Script.COMMON_TURKIC)
print(t.transliterate("Azərbaycan dili"))
# → Azärbaycan dili

t = Transliterator("tuk", Script.LATIN, Script.COMMON_TURKIC)
print(t.transliterate("Türkmen dili baýdyr"))
# → Türkmen dili baydır
```

#### Old Turkic Runic Script

TurkicNLP supports transliteration of [Old Turkic runic inscriptions](https://en.wikipedia.org/wiki/Old_Turkic_script) (Orkhon-Yenisei script, Unicode block U+10C00–U+10C4F) to Latin:

```python
from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator

t = Transliterator("otk", Script.OLD_TURKIC_RUNIC, Script.LATIN)

# Individual runic characters
print(t.transliterate("\U00010C45\U00010C07\U00010C3C\U00010C1A"))  # 𐱅𐰇𐰼𐰚
# → törk  (Türk)

# The transliterator maps each runic character to its standard
# Turkological Latin equivalent, handling both Orkhon and Yenisei
# variant forms (e.g., separate glyphs for consonants with
# back vs. front vowel contexts).
```

### Neural POS Tagger & Dependency Parser (Glot500)

The multilingual Glot500-based model provides UPOS tagging and dependency parsing for 15 Turkic languages (10 trained + 5 zero-shot). Requires `pip install "turkicnlp[transformers]"`.

```python
import turkicnlp

# Download tokenizer + multilingual Glot500 POS/DepParse model
turkicnlp.download("kaz", processors=["tokenize", "pos", "depparse"])

nlp = turkicnlp.Pipeline(
    "kaz",
    processors=["tokenize", "pos", "depparse"],
    pos_backend="multilingual_glot500",
    depparse_backend="multilingual_glot500",
)

doc = nlp("Мен мектепке бардым.")

for sentence in doc.sentences:
    for word in sentence.words:
        print(f"{word.text:12} {word.upos:6} head={word.head} {word.deprel}")
```

### Morpheme Tokenizer (Hybrid FST + Neural)

The `MorphemeTokenizer` segments inflected Turkic words into labelled surface morphemes. For every word it converts
each Apertium HFST reading into a segmentation by matching the reading's tags against language-specific suffix
allomorph tables (vowel harmony, consonant alternations, buffer consonants), and keeps the reading that explains the
whole word, breaking ties by agreement with the neural (Glot500) lemma and part of speech. Derivational suffixes are
split only when the transducer confirms the shorter stem. When the transducer has no usable reading, the neural
analysis is segmented instead. Requires `pip install "turkicnlp[hfst,transformers]"` (works with only one of the
two, with lower accuracy).

On a 483-word test set covering 13 word categories in 20 languages, the hybrid segments 89% of the words exactly
(Apertium only: 79%, neural only: 68%). The test set and the evaluation script are in the paper's companion
repository; segmentation is weakest for derivation and in languages with small Apertium lexicons
(see [ROADMAP.md](ROADMAP.md#2-known-issues)).

```python
from turkicnlp.processors.morpheme_tokenizer import MorphemeTokenizer

# --- Kazakh ---
tok = MorphemeTokenizer(lang="kaz")
tok.load()

result = tok.segment("бармадым")
print(result.segments)
# ['бар', 'ма', 'ды', 'м']

print(result.labeled)
# [('бар', 'STEM'), ('ма', 'NEG'), ('ды', 'PST'), ('м', '1SG')]

result = tok.segment("оқығандар")
print(result.labeled)
# [('оқы', 'STEM'), ('ған', 'PTCP.PST'), ('дар', 'PLUR')]

# --- Turkish ---
tok = MorphemeTokenizer(lang="tur")
tok.load()

result = tok.segment("evlerinden")
print(result.labeled)
# [('ev', 'STEM'), ('ler', 'PLUR'), ('in', 'POSS.2SG'), ('den', 'ABL')]

result = tok.segment("gidiyorlar")
print(result.labeled)
# [('gid', 'STEM'), ('iyor', 'PROG'), ('lar', '3PL')]

# Apostrophe boundaries are handled for proper nouns
result = tok.segment("İstanbul'da")
print(result.labeled)
# [('İstanbul', 'STEM'), ("'da", 'LOC')]

# --- Uzbek ---
tok = MorphemeTokenizer(lang="uzb")
tok.load()

result = tok.segment("kitobimdan")
print(result.labeled)
# [('kitob', 'STEM'), ('im', 'POSS.1SG'), ('dan', 'ABL')]

result = tok.segment("bolalarning")
print(result.labeled)
# [('bola', 'STEM'), ('lar', 'PLUR'), ('ning', 'GEN')]
```

Suffix allomorph tables exist for 23 languages (`turkicnlp/resources/morpheme_rules.json`; all except Old Turkic); the Apertium path is
available for the 20 languages with a transducer. Results are cached per word (`MorphemeTokenizer(lang, cache_size=...)`).

### Morpheme-aware subword tokenization (`sp_tokenize`)

`sp_tokenize` encodes text with a SentencePiece model trained on morpheme-segmented text. Input words are segmented
with the `MorphemeTokenizer` first, so pieces do not cross morpheme boundaries. It adds `sp_tokens` and `sp_ids` to
every sentence. Pretrained models are not released yet ([ROADMAP.md](ROADMAP.md#5-tokenization-for-language-models));
train one on your corpus with `turkicnlp/tools/annotate_corpus.py` + SentencePiece and pass its path:

```python
import turkicnlp

nlp = turkicnlp.Pipeline(
    "kaz",
    processors=["tokenize", "sp_tokenize"],
    sp_tokenize_model_path="models/sp/sp_kaz.model",
)
doc = nlp("Мен мектептерімізден оқыдым")
print(doc.sentences[0].sp_tokens)
```

### Neural Morphological Analyzer & Lemmatizer (Glot500)

The multilingual Glot500-based morph model provides UPOS tagging, UD morphological features, and lemmatization for 23 Turkic languages. Requires `pip install "turkicnlp[transformers]"`.

```python
import turkicnlp

# Download tokenizer + multilingual Glot500 morph model
turkicnlp.download("tur", processors=["tokenize", "morph_neural"])

nlp = turkicnlp.Pipeline(
    "tur",
    processors=["tokenize", "morph_neural"],
)

doc = nlp("Çocuklar okula gidiyorlar.")

for sentence in doc.sentences:
    for word in sentence.words:
        print(f"{word.text:20} {word.upos:6} {word.lemma:15} {word.feats}")
```

Output:
```
Çocuklar             NOUN   çocuk           Case=Nom|Number=Plur
okula                NOUN   okul            Case=Dat|Number=Sing
gidiyorlar           VERB   gitmek          Aspect=Prog|Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin
.                    PUNCT  .               _
```

The morph analyzer also works for low-resource languages:

```python
# Sakha (Yakut) — directly trained
turkicnlp.download("sah", processors=["tokenize", "morph_neural"])
nlp = turkicnlp.Pipeline("sah", processors=["tokenize", "morph_neural"])
doc = nlp("Мин оскуолаҕа бардым.")

# Karakalpak — zero-shot via Uzbek proxy embedding
turkicnlp.download("kaa", processors=["tokenize", "morph_neural"])
nlp = turkicnlp.Pipeline("kaa", processors=["tokenize", "morph_neural"])
doc = nlp("Men mektepke bardım.")
```

## Supported Languages and Components

<p align="center">
  <img src="https://upload.wikimedia.org/wikipedia/commons/f/fa/Turkic_Languages_distribution_map.png" alt="Distribution map of Turkic languages" width="700">
  <br>
  <em>Geographic distribution of Turkic languages (source: <a href="https://commons.wikimedia.org/wiki/File:Turkic_Languages_distribution_map.png">Wikimedia Commons</a>)</em>
</p>

The table below shows all supported languages with their available scripts and processor status.

**Legend:**

| Symbol | Backend | Description |
|:---:|---|---|
| ■ | Rule-based | Regex tokenizer, abbreviation lists |
| ◆ | [Apertium](https://apertium.org/) FST | Finite-state morphology via `hfst` ([GPL-3.0](https://www.gnu.org/licenses/gpl-3.0.html), downloaded separately) |
| ● | [Stanza/UD](https://stanfordnlp.github.io/stanza/) | Neural models trained on [Universal Dependencies](https://universaldependencies.org/) treebanks |
| ▲ | Custom Stanza | Custom-trained Stanza models hosted by [turkic-nlp](https://github.com/turkic-nlp/trained-stanza-models) |
| ◇ | [Glot500](https://github.com/cisnlp/Glot500) Neural | Multilingual POS tagger & dependency parser (Glot500 backbone, 15 languages) |
| ◈ | [Glot500](https://github.com/cisnlp/Glot500) Neural Morph | Multilingual morphological analyzer & lemmatizer (Glot500 backbone, 22 languages) |
| ★ | [NLLB](https://huggingface.co/facebook/nllb-200-distilled-600M) | Embeddings and machine translation via NLLB-200 |
| ◎ | [Omnilingual ASR](https://github.com/facebookresearch/omnilingual-asr) | Speech recognition (`SpeechRecognizer`, speech + text environment) |
| ○ | Planned | Implementation planned |
| — | | Not available yet |

### Oghuz Branch

| Language | Code | Script(s) | Tokenize | Morph | POS | Lemma | DepParse | NER | Embed | Translate | ASR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| [Turkish](https://en.wikipedia.org/wiki/Turkish_language) | `tur` | Latn | ■ ● | ◆ ◈ | ● ◇ | ● ◈ | ● ◇ | ● | ★ | ★ | ◎ |
| [Azerbaijani](https://en.wikipedia.org/wiki/Azerbaijani_language) | `aze` | Latn, Cyrl | ■▲ | ◆ ◈ | ▲ ◇ | ▲ ◈ | ▲ ◇ | — | ★ | ★ | ◎ |
| [Iranian Azerbaijani](https://en.wikipedia.org/wiki/South_Azerbaijani_language) | `azb` | Arab | ■ | — | — | — | — | — | ★ | ★ | ◎ |
| [Turkmen](https://en.wikipedia.org/wiki/Turkmen_language) | `tuk` | Latn, Cyrl | ■▲ | ◆ ◈ | ▲ ◇ | ▲ ◈ | ▲ ◇ | — | ★ | ★ | ◎ |
| [Gagauz](https://en.wikipedia.org/wiki/Gagauz_language) | `gag` | Latn | ■ | ◆ ◈ | ◈ | ◈ | — | — | — | — | ◎ |

### Kipchak Branch

| Language | Code | Script(s) | Tokenize | Morph | POS | Lemma | DepParse | NER | Embed | Translate | ASR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| [Kazakh](https://en.wikipedia.org/wiki/Kazakh_language) | `kaz` | Cyrl, Latn | ■ ● | ◆ ◈ | ● ◇ | ● ◈ | ● ◇ | ● | ★ | ★ | ◎ |
| [Kyrgyz](https://en.wikipedia.org/wiki/Kyrgyz_language) | `kir` | Cyrl | ■ ● | ◆ ◈ | ● ◇ | ● ◈ | ● ◇ | — | ★ | ★ | ◎ |
| [Tatar](https://en.wikipedia.org/wiki/Tatar_language) | `tat` | Cyrl, Latn | ■▲ | ◆ ◈ | ▲ ◇ | ▲ ◈ | ▲ ◇ | — | ★ | ★ | ◎ |
| [Bashkir](https://en.wikipedia.org/wiki/Bashkir_language) | `bak` | Cyrl | ■▲ | ◆ ◈ | ▲ ◇ | ▲ ◈ | ▲ ◇ | — | ★ | ★ | ◎ |
| [Crimean Tatar](https://en.wikipedia.org/wiki/Crimean_Tatar_language) | `crh` | Latn, Cyrl | ■ | ◆ ◈ | ◈ | ◈ | — | — | ★ | ★ | ◎ |
| [Karakalpak](https://en.wikipedia.org/wiki/Karakalpak_language) | `kaa` | Latn, Cyrl | ■ | ◆ ◈ | ◇ ◈ | ◈ | ◇ | — | — | — | ◎ |
| [Nogai](https://en.wikipedia.org/wiki/Nogai_language) | `nog` | Cyrl | ■ | ◆ ◈ | ◇ ◈ | ◈ | ◇  | — | — | — | ◎ |
| [Kumyk](https://en.wikipedia.org/wiki/Kumyk_language) | `kum` | Cyrl | ■ | ◆ ◈ | ◇ ◈ | ◈ | ◇ | — | — | — | ◎ |
| [Karachay-Balkar](https://en.wikipedia.org/wiki/Karachay-Balkar_language) | `krc` | Cyrl | ■ | ◆ ◈ | ◇ ◈ | ◈ | ◇ | — | — | — | ◎ |

### Karluk Branch

| Language | Code | Script(s) | Tokenize | Morph | POS | Lemma | DepParse | NER | Embed | Translate | ASR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| [Uzbek](https://en.wikipedia.org/wiki/Uzbek_language) | `uzb` | Latn, Cyrl | ■ ▲ | ◆ ◈ | ▲ ◇ | ▲ ◈ | ▲ ◇ | — | ★ | ★ | ◎ |
| [Uyghur](https://en.wikipedia.org/wiki/Uyghur_language) | `uig` | Arab, Latn, Cyrl | ■ ● | ◆ ◈ | ● ◇ | ● ◈ | ● ◇ | — | ★ | ★ | ◎ |

### Siberian Branch

| Language | Code | Script(s) | Tokenize | Morph | POS | Lemma | DepParse | NER | Embed | Translate | ASR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| [Sakha (Yakut)](https://en.wikipedia.org/wiki/Sakha_language) | `sah` | Cyrl | ■ | ◆ ◈ | ◇ ◈ | ◈ | ◇ | — | — | — | ◎ |
| [Altai](https://en.wikipedia.org/wiki/Altai_language) | `alt` | Cyrl | ■ | ◆ ◈ | ◈ | ◈ | — | — | — | — | ◎ |
| [Tuvan](https://en.wikipedia.org/wiki/Tuvan_language) | `tyv` | Cyrl | ■ | ◆ ◈ | ◈ | ◈ | — | — | — | — | — |
| [Khakas](https://en.wikipedia.org/wiki/Khakas_language) | `kjh` | Cyrl | ■ | ◆ ◈ | ◈ | ◈ | — | — | — | — | ◎ |

### Oghur Branch

| Language | Code | Script(s) | Tokenize | Morph | POS | Lemma | DepParse | NER | Embed | Translate | ASR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| [Chuvash](https://en.wikipedia.org/wiki/Chuvash_language) | `chv` | Cyrl | ■ | ◆ ◈ | ◈ | ◈ | — | — | — | — | ◎ |

### Arghu Branch

| Language | Code | Script(s) | Tokenize | Morph | POS | Lemma | DepParse | NER | Embed | Translate | ASR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| [Khalaj](https://en.wikipedia.org/wiki/Khalaj_language) | `klj` | Latn | ■ | ◈ | ◈ | ◈ | — | — | — | — | — |

### Historical Languages

| Language | Code | Script(s) | Tokenize | Morph | POS | Lemma | DepParse | NER | Embed | Translate | ASR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| [Ottoman Turkish](https://en.wikipedia.org/wiki/Ottoman_Turkish_language) | `ota` | Arab, Latn | ■ | ◈ | ◇ ◈ | ◈ | ◇ | — | — | — | — |
| [Old Turkic](https://en.wikipedia.org/wiki/Old_Turkic_language) | `otk` | Orkh, Latn | ■ | — | — | — | — | — | — | — | — |

### Stanza/UD Model Details

The Stanza backend provides neural models trained on [Universal Dependencies](https://universaldependencies.org/) treebanks. Official Stanza models (●) are downloaded via Stanza's model hub. Custom-trained models (▲) are hosted at [turkic-nlp/trained-stanza-models](https://github.com/turkic-nlp/trained-stanza-models) and downloaded automatically.

| Language | Stanza Code | Type | UD Treebank(s) | Stanza Processors | NER Dataset |
|---|---|---|---|---|---|
| Turkish | `tr` | ● | IMST (default), BOUN, FrameNet, KeNet, ATIS, Penn, Tourism | tokenize, pos, lemma, depparse, ner | Starlang NER |
| Kazakh | `kk` | ● | KTB | tokenize, pos, lemma, depparse, ner | KazNERD |
| Uyghur | `ug` | ● | UDT | tokenize, pos, lemma, depparse | — |
| Kyrgyz | `ky` | ● | KTMU | tokenize, pos, lemma, depparse | — |
| Uzbek | `uz` | ▲ | UzUDT | tokenize, pos, lemma, depparse | — |
| Turkmen | `tk` | ▲ | [Tk-TUD](https://github.com/turkic-nlp/generated-ud-data/tree/main/tuk) | tokenize, pos, lemma, depparse | — |
| Azerbaijani | `az` | ▲ | [Az-TUD](https://github.com/turkic-nlp/generated-ud-data/tree/main/aze) | tokenize, pos, lemma, depparse | — |
| Tatar | `tt` | ▲ | [Ta-TUD](https://github.com/turkic-nlp/generated-ud-data/tree/main/tat) | tokenize, pos, lemma, depparse | — |
| Bashkir | `ba` | ▲ | [Ba-TUD](https://github.com/turkic-nlp/generated-ud-data/tree/main/bak) | tokenize, pos, lemma, depparse | — |

### Multilingual Glot500 Neural Models

TurkicNLP provides two multilingual neural models built on a frozen [Glot500](https://github.com/cisnlp/Glot500) backbone with script adapters, language embeddings, and shared BiLSTM layers. Both models are hosted at [turkic-nlp/trained-stanza-models](https://github.com/turkic-nlp/trained-stanza-models/releases) and downloaded automatically.

| Model | Symbol | Tasks | Languages | Architecture |
|---|:---:|---|---|---|
| POS & DepParser | ◇ | UPOS, dependency parsing | 10 trained + 5 zero-shot (15 total) | Glot500 → ScriptAdapter → LangEmbed → BiLSTM → POS Head + Biaffine Parser |
| Morph Analyzer | ◈ | UPOS, UD morph features, lemmatization | 20 trained + 3 zero-shot (23 model languages = 22 TurkicNLP codes; both Altai varieties map to `alt`) | Glot500 → ScriptAdapter → LangEmbed → BiLSTM → POS Head + MorphFeat Head + CharCNN LemmaHead |

**POS & DepParser supported languages:** Turkish, Azerbaijani, Uzbek, Turkmen, Kazakh, Kyrgyz, Bashkir, Tatar, Uyghur, Ottoman Turkish + zero-shot: Karakalpak, Kumyk, Sakha, Karachay-Balkar, Nogai

**Morph Analyzer supported languages:** Turkish, Azerbaijani, Uzbek, Turkmen, Kazakh, Kyrgyz, Bashkir, Tatar, Uyghur, Ottoman Turkish, Crimean Tatar, Khakas, Sakha, Tuvan, Chuvash, Gagauz, Kumyk, Southern Altai, Khalaj, Northern Altai + zero-shot: Karakalpak, Karachay-Balkar, Nogai

### Transliteration Support

Bidirectional script conversion is available for all multi-script languages. The transliterator uses a greedy longest-match algorithm with per-language mapping tables.

| Language | Direction | Scripts | Standard |
|---|---|---|---|
| Kazakh | ↔ Bidirectional | Cyrillic ↔ Latin | 2021 official Latin alphabet |
| Uzbek | ↔ Bidirectional | Cyrillic ↔ Latin | 1995 official Latin alphabet |
| Azerbaijani | ↔ Bidirectional | Cyrillic ↔ Latin | 1991 official Latin alphabet |
| Tatar | ↔ Bidirectional | Cyrillic ↔ Latin | Zamanälif |
| Turkmen | ↔ Bidirectional | Cyrillic ↔ Latin | 1993 official Latin alphabet |
| Karakalpak | ↔ Bidirectional | Cyrillic ↔ Latin | 2016 Latin alphabet |
| Crimean Tatar | ↔ Bidirectional | Cyrillic ↔ Latin | Standard Crimean Tatar Latin |
| Uyghur | ↔ Bidirectional | Perso-Arabic ↔ Latin, Perso-Arabic ↔ Cyrillic, Latin ↔ Cyrillic | Uyghur Latin Yéziqi (ULY), Uyghur Cyrillic |
| Ottoman Turkish | → One-way | Latin → Perso-Arabic | Academic transcription |
| Old Turkic | → One-way | Runic → Latin | Turkological convention |
| All except Old Turkic (23) | → native script to CTS | Latin / Cyrillic / Perso-Arabic → CTS | [Common Turkic Alphabet](https://en.wikipedia.org/wiki/Common_Turkic_alphabet) |

### Apertium FST Quality Levels

| Level | Description | Languages |
|---|---|---|
| **Production** | >90% coverage on news text | Turkish, Kazakh, Tatar |
| **Stable** | Good coverage, actively maintained | Kyrgyz, Uzbek |
| **Beta** | Reasonable coverage, some gaps | Azerbaijani (many common verbs missing), Turkmen, Bashkir, Uyghur, Crimean Tatar, Chuvash |
| **Prototype** | Limited coverage, experimental | Gagauz, Sakha, Karakalpak, Nogai, Kumyk, Karachay-Balkar, Altai, Tuvan, Khakas |

### Model Catalog and Apertium Downloads

TurkicNLP uses a model catalog to define download sources per language/script/processor. The catalog lives in:

- `turkicnlp/resources/catalog.json` (packaged default)
- Remote override: `ModelRegistry.CATALOG_URL` (or `TURKICNLP_CATALOG_URL`)

For each language and script, the catalog lists the available processors, their backends, the default backend and
the download source. `turkicnlp.download()` reads it and fetches what is missing: precompiled Apertium `.hfst`
transducers from the [turkic-nlp/apertium-data](https://github.com/turkic-nlp/apertium-data) releases (built by CI
from the upstream Apertium repositories), custom Stanza and Glot500 checkpoints from
[turkic-nlp/trained-stanza-models](https://github.com/turkic-nlp/trained-stanza-models/releases), and Hugging Face
models (Glot500 backbone, NLLB). When a release contains several transducers (e.g. `krc@Seegmiller`, `uzb_guesser`),
the main `<lang>.automorf.hfst` is used.

#### Download folder
All models and resources are downloaded into this folder: `~/.turkicnlp`.


## Architecture

TurkicNLP follows Stanza's modular pipeline design:

```
Pipeline("tur", processors=["tokenize", "morph", "pos", "ner", "depparse"])
    │
    ▼
  Document ─── text: "Ben okula vardım"
    │
    ├── script_detect    → script = "Latn"
    ├── tokenize         → sentences, tokens, words
    ├── morph (Apertium) → lemma, pos, feats (via HFST)
    ├── pos (Stanza)     → UPOS, XPOS, feats (owns these fields)
    ├── ner (Stanza)     → BIO tags and entity spans
    └── depparse         → head, deprel
    │
    ▼
  Document ─── annotated with all layers
```

```
Pipeline("sah", processors=["tokenize", "morph_neural"])
    │
    ▼
  Document ─── text: "Мин оскуолаҕа бардым"
    │
    ├── script_detect          → script = "Cyrl"
    ├── tokenize               → sentences, tokens, words
    └── morph_neural (Glot500) → upos, feats, lemma
    │
    ▼
  Document ─── annotated with morphological analysis
```

```
Pipeline("azb", processors=["embeddings", "translate"], translate_tgt_lang="eng")
    │
    ▼
  Document ─── text: "من کتاب اوخویورام"
    │
    ├── script_detect          → script = "Arab"
    ├── embeddings (NLLB)      → sentence/document vectors
    └── translate (NLLB)       → sentence/document translation
           (src resolved from FLORES map: azb -> azb_Arab,
            tgt resolved from ISO-3: eng -> eng_Latn)
    │
    ▼
  Document ─── annotated with all layers
```

### Processors and backends

Use these names in `Pipeline(lang, processors=[...])`; pick a backend with `<name>_backend="..."` and pass options
as `<name>_<option>=...` (e.g. `translate_tgt_lang="eng"`). Dependencies are added automatically.

| Processor | Backends (default first where the catalog has several) | Adds | Loaded by default |
|---|---|---|---|
| `tokenize` | `rule`, `rule_arabic`, `stanza` | sentences, tokens, words | ✓ |
| `mwt` | `neural` (rule-based), `stanza` | multi-word token expansion | ✓ (where available) |
| `morph` | `apertium`, `llm`, `apertium_llm` | lemma, UPOS, feats (HFST / LLM / HFST + LLM choice) | ✓ (`apertium`) |
| `morph_neural` | `multilingual_glot500_morph` | UPOS, feats, lemma | ✓ if no `morph` |
| `pos` | `stanza`, `multilingual_glot500_model` | UPOS, XPOS, feats | ✓ |
| `feats` | `multilingual_glot500_morph` | UD features | ✓ |
| `lemma` | `stanza`, `multilingual_glot500_morph` | lemma | ✓ |
| `depparse` | `stanza`, `multilingual_glot500_model` | head, deprel | ✓ |
| `ner` | `stanza` (tur, kaz), `llm` (all) | BIO tags, entity spans | ✓ (`stanza`) |
| `embeddings` | `nllb` | sentence/document vectors | – |
| `translate` | `nllb`, `llm` | sentence/document translation | – |
| `sentiment` | `llm` | sentence sentiment label | – |
| `spellcheck` | `llm` | `sentence.correction`, `doc.corrected_text` | – |
| `sp_tokenize` | `sentencepiece` | `sp_tokens`, `sp_ids` | – |
| `asr` | Omnilingual ASR (via `Pipeline.from_audio`) | transcript | – |

`turkicnlp.list_processors("kaz")` lists what the catalog offers for a language; the `llm` backends work for every
language and are not listed there. `sentiment` without a backend option uses `llm`. The placeholder backends
`tokenize=neural`, `morph=neural`, `pos=neural`, `lemma=neural`, `lemma=dictionary`, `depparse=neural` and
`sentiment=neural` are registered but not implemented; the pipeline never selects them by itself.

**Which processor writes which field.** Each word field has one owner per pipeline: the last processor in
execution order that provides it (`pos` owns UPOS/XPOS, `feats` FEATS, `lemma` LEMMA, `depparse` HEAD/DEPREL; a
full analyser such as `morph` owns the fields that no later step provides). In the default Turkish pipeline
(`morph`, `pos`, `lemma`, `depparse`) Stanza therefore provides UPOS, FEATS and LEMMA. What other processors wrote
is kept per word in `word._provenance` (e.g. `word._provenance["morph"]` holds the Apertium lemma, UPOS and FEATS),
and FEATS from a different processor than UPOS are dropped for a word when that processor's own UPOS disagrees.
When `morph=apertium` is in the default set, the Glot500 `feats`/`lemma` steps of the same language are left out.

**Scripts without models.** If a language has no models for the input script (e.g. Uzbek Cyrillic, Kazakh
Latin), the pipeline transliterates the text to the script that has them and maps the results back: word texts and
character offsets refer to the original input, lemmas are transliterated back, and `doc._model_text` holds the
transliterated text.

### Key Abstractions

- **Document** → Sentence → Token → Word hierarchy (maps to CoNLL-U)
- **Processor** ABC with `PROVIDES`, `REQUIRES`, `NAME` class attributes
- **Pipeline** orchestrator with dependency resolution and script-aware model loading
- **ProcessorRegistry** for pluggable backends (rule, Apertium, Stanza, Glot500, NLLB)
- **ModelRegistry** with remote catalog and local caching at `~/.turkicnlp/models/`
- **NLLB FLORES language map** for ISO-3 to NLLB code resolution in translation (e.g. `tuk` -> `tuk_Latn`)

### Model Storage Layout

```
~/.turkicnlp/models/
├── kaz/
│   ├── Cyrl/
│   │   ├── tokenize/rule/
│   │   └── morph/apertium/    ← GPL-3.0 (downloaded separately)
│   │       ├── kaz.automorf.hfst
│   │       ├── LICENSE
│   │       └── metadata.json
│   └── ...
├── tur/
│   └── Latn/
│       └── ...
├── multilingual/
│   ├── multilingual_glot500.pt           ← POS/DepParse checkpoint
│   └── multilingual_morph_glot500.pt     ← Morph analyzer checkpoint
├── huggingface/
│   ├── cis-lmu--glot500-base/            ← Shared Glot500 backbone
│   │   ├── config.json
│   │   ├── model.safetensors
│   │   └── ...
│   └── facebook--nllb-200-distilled-600M/
│       ├── config.json
│       ├── model.safetensors (or pytorch_model.bin)
│       ├── tokenizer.json
│       └── ...
└── stanza/, stanza_custom/                ← Stanza models

# Stanza models (official and custom) are stored under ~/.turkicnlp/models/stanza*/
```

Notes:
- Omnilingual ASR checkpoints are managed by fairseq2 and cached in `~/.cache/fairseq2/assets/`.
- NLLB embeddings and translation use a shared Hugging Face model under `~/.turkicnlp/models/huggingface/`.
- The NLLB model is downloaded once and reused across supported Turkic languages.
- The Glot500 backbone is shared between the POS/DepParse and Morph analyzer models under `~/.turkicnlp/models/huggingface/`.
- Unlike Apertium/Stanza components, NLLB and Glot500 artifacts are not duplicated per language/script directory.

## License

- **Library code**: [Apache License 2.0](LICENSE)
- **Stanza models**: [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) — managed by Stanza's own download mechanism
- **Apertium FST data**: [GPL-3.0](https://www.gnu.org/licenses/gpl-3.0.html) — downloaded separately at runtime, never bundled in the pip package
- **Omnilingual ASR models and code**: [Apache License 2.0](https://github.com/facebookresearch/omnilingual-asr/blob/main/LICENSE) — downloaded by fairseq2 at runtime
- **NLLB-200 model weights/tokenizer**: [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/) — downloaded from Hugging Face at runtime and reused from `~/.turkicnlp/models/huggingface/` (non-commercial license terms apply)

## Development

```bash
git clone https://github.com/turkic-nlp/turkicnlp.git
cd turkicnlp
pip install -e ".[dev]"
python -m pytest        # "python -m" guarantees the pytest of the active environment

# Speech recognition tests run without the model (the backend is mocked);
# run the full suite in the speech + text environment as well:
source venv-asr/bin/activate && pip install pytest && python -m pytest
```

## Contributing

Contributions are very welcome — from checking analyses in your language to adding new components. Read
**[CONTRIBUTING.md](CONTRIBUTING.md)** for the development setup, how to add a language or a processor, tests, code
style and the pull-request process, and browse **[ROADMAP.md](ROADMAP.md)** for open tasks (items marked **S** are
good first issues). Questions are welcome on [Discord](https://discord.gg/CeVTbGpmMQ) or in
[GitHub issues](https://github.com/turkic-nlp/turkicnlp/issues).



## Acknowledgements

TurkicNLP builds on the work of many researchers and communities. We gratefully acknowledge the following:

### Stanza

[Stanza](https://stanfordnlp.github.io/stanza/) provides the pretrained neural models for tokenization, POS tagging, lemmatization, dependency parsing, and NER.

> Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. *Stanza: A Python Natural Language Processing Toolkit for Many Human Languages*. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. [[paper]](https://aclanthology.org/2020.acl-demos.14/)

### Universal Dependencies Treebanks

The Stanza models are trained on [Universal Dependencies](https://universaldependencies.org/) treebanks created by the following teams:

**Turkish (UD_Turkish-IMST)**
> Umut Sulubacak, Memduh Gokirmak, Francis Tyers, Cagri Coltekin, Joakim Nivre, and Gulsen Cebiroglu Eryigit. *Universal Dependencies for Turkish*. COLING 2016. [[paper]](https://aclanthology.org/C16-1325/)

**Turkish (UD_Turkish-BOUN)**
> Utku Turk, Furkan Atmaca, Saziye Betul Ozates, Gozde Berk, Seyyit Talha Bedir, Abdullatif Koksal, Balkiz Ozturk Basaran, Tunga Gungor, and Arzucan Ozgur. *Resources for Turkish Dependency Parsing: Introducing the BOUN Treebank and the BoAT Annotation Tool*. Language Resources and Evaluation 56(1), 2022. [[paper]](https://doi.org/10.1007/s10579-021-09558-0)

**Turkish (UD_Turkish-FrameNet, KeNet, ATIS, Penn, Tourism)**
> Busra Marsan, Neslihan Kara, Merve Ozcelik, Bilge Nas Arican, Neslihan Cesur, Asli Kuzgun, Ezgi Saniyar, Oguzhan Kuyrukcu, and Olcay Taner Yildiz. [Starlang Software](https://starlangyazilim.com/) and [Ozyegin University](https://www.ozyegin.edu.tr/). These treebanks cover diverse domains including FrameNet frames, WordNet examples, airline travel, Penn Treebank translations, and tourism reviews.

**Kazakh (UD_Kazakh-KTB)**
> Aibek Makazhanov, Jonathan North Washington, and Francis Tyers. *Towards a Free/Open-source Universal-dependency Treebank for Kazakh*. TurkLang 2015. [[paper]](https://universaldependencies.org/treebanks/kk_ktb/)

**Uyghur (UD_Uyghur-UDT)**
> Marhaba Eli (Xinjiang University), Daniel Zeman (Charles University), and Francis Tyers. [[treebank]](https://universaldependencies.org/treebanks/ug_udt/)

**Kyrgyz (UD_Kyrgyz-KTMU)**
> Ibrahim Benli. [[treebank]](https://universaldependencies.org/treebanks/ky_ktmu/)

**Ottoman Turkish (UD_Ottoman_Turkish-BOUN)**
> Saziye Betul Ozates, Tarik Emre Tiras, Efe Eren Genc, and Esma Fatima Bilgin Tasdemir. *Dependency Annotation of Ottoman Turkish with Multilingual BERT*. LAW-XVIII, 2024. [[paper]](https://aclanthology.org/2024.law-1.18)

### NER Datasets

**Turkish NER (Starlang)**
> B. Ertopcu, A. B. Kanburoglu, O. Topsakal, O. Acikgoz, A. T. Gurkan, B. Ozenc, I. Cam, B. Avar, G. Ercan, and O. T. Yildiz. *A New Approach for Named Entity Recognition*. UBMK 2017. [[paper]](https://doi.org/10.1109/UBMK.2017.8093439)

**Kazakh NER (KazNERD)**
> Rustem Yeshpanov, Yerbolat Khassanov, and Huseyin Atakan Varol (ISSAI, Nazarbayev University). *KazNERD: Kazakh Named Entity Recognition Dataset*. LREC 2022. [[paper]](https://aclanthology.org/2022.lrec-1.44)

### Glot500

The multilingual Glot500 model serves as the frozen backbone for TurkicNLP's neural POS/DepParse and Morph analyzer models.

> ImaniGooghari, Ayyoob, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. 2023. *Glot500: Scaling Multilingual Corpora and Language Technology to 500 Languages*. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [[paper]](https://aclanthology.org/2023.acl-long.61/)

### GlotLID

The language detection model

> Amir Hossein Kargaran, Ayyoob Imani, François Yvon, Hinrich Schütze (2023). GlotLID: Language Identification for Low-Resource Languages. The 2023 Conference on Empirical Methods in Natural Language Processing. [Model](https://huggingface.co/cis-lmu/glotlid)


### Wiktextract / Kaikki.org

Morphological training data for extended Turkic languages was extracted from Wiktionary using Wiktextract. The structured data is available at [kaikki.org](https://kaikki.org/).

> Tatu Ylonen. 2022. *Wiktextract: Wiktionary as Machine-Readable Structured Data*. In Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022). [[paper]](https://aclanthology.org/2022.lrec-1.140/)

### UniMorph

The [Universal Morphology (UniMorph)](https://unimorph.github.io/) project provides morphological paradigms used for training and evaluating the multilingual morph analyzer across Turkic languages.

> John Sylak-Glassman. 2016. *The Composition and Use of the Universal Morphological Feature Schema (UniMorph Schema)*. Johns Hopkins University. [[paper]](https://unimorph.github.io/doc/unimorph-schema.pdf)

### NLLB Embeddings & Machine Translation

TurkicNLP embeddings backend uses encoder pooling on:

> [facebook/nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M)

Reference:
> NLLB Team, Marta R. Costa-jussà, et al. 2022. *No Language Left Behind: Scaling Human-Centered Machine Translation*. [[paper]](https://arxiv.org/abs/2207.04672)

### Omnilingual ASR

Speech recognition uses Meta's Omnilingual ASR models:

> [facebookresearch/omnilingual-asr](https://github.com/facebookresearch/omnilingual-asr), [facebook/omniASR-LLM-1B](https://huggingface.co/facebook/omniASR-LLM-1B)

Reference:
> Omnilingual ASR Team. 2025. *Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages*.

### Common Turkic Alphabet

> [Common Turkic Alphabet](https://en.wikipedia.org/wiki/Common_Turkic_alphabet)


### Uyghur Transliteration

> [Uyghur-Multi-Script-Converter](https://github.com/neouyghur/Uyghur-Multi-Script-Converter)

Reference:
> Osman, T. Uyghur Multi-Script Converter (Version 1.0) [Computer software]


### Other Organisations

- [Apertium](https://apertium.org/) — morphological transducers covering 20+ Turkic languages
- [SIGTURK](https://sigturk.github.io/) — ACL Special Interest Group on Turkic Languages
- [ISSAI](https://issai.nu.edu.kz/) — Institute of Smart Systems and Artificial Intelligence, Nazarbayev University, for Kazakh NLP resources
- [Universal Dependencies](https://universaldependencies.org/) — the framework and community behind Turkic treebanks
- [Turkic Interlingua](https://github.com/turkic-interlingua) — resources for machine translation for Turkic languages
- [Turkic UD](https://github.com/ud-turkic) - group working on harmonizing Turkic UD treebanks
- [TurkLang](https://web.archive.org/web/20260117063647/https://turklang.ieees.org/) - Conference on Computer Processing of Turkic Languages (2013-present)
- [Turkic Academy](https://turkicacademy.org/en) - facilitates and encourages scientific and educational cooperation among Turkic countries and communities.
- [Organization of Turkic States](https://turkicstates.org/en) - primary objective is to foster comprehensive cooperation among Turkic States
