Metadata-Version: 2.4
Name: turkic-translit
Version: 0.3.9
Summary: Deterministic Latin and IPA transliteration for Kazakh, Kyrgyz, Uzbek, Turkish, Azerbaijani, Uyghur, Finnish, plus tokenizer/glue scripts.
Author-email: Austin Wagner <austinwagner@msn.com>
License-Expression: MIT
Keywords: kazakh,kyrgyz,transliteration,ipa
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: epitran<1.27,>=1.0
Requires-Dist: fasttext-wheel==0.9.2
Requires-Dist: numpy<2
Requires-Dist: packaging>=23.0
Requires-Dist: panphon<0.22,>=0.20
Requires-Dist: PyICU<2.16,>=2.13; sys_platform != "win32"
Requires-Dist: pytest>=8.0
Requires-Dist: rapidfuzz>=3.5
Requires-Dist: rich>=13.7
Requires-Dist: python-json-logger>=2.0.4
Requires-Dist: sentencepiece>=0.2.0
Requires-Dist: click>=8.1
Requires-Dist: types-requests>=2.31
Requires-Dist: datasets<4.0.0,>=3.6.0
Requires-Dist: evaluate>=0.4
Requires-Dist: accelerate>=0.28
Requires-Dist: transformers<5,>=4.41
Requires-Dist: tqdm>=4.66
Requires-Dist: scikit-learn>=1.4
Requires-Dist: typing_extensions>=4.12
Requires-Dist: wikipedia<2.0.0,>=1.4.0
Requires-Dist: gradio>=4.0
Requires-Dist: pandas
Requires-Dist: matplotlib>=3.9
Requires-Dist: pycountry>=23.12
Requires-Dist: zstandard>=0.23
Provides-Extra: examples
Requires-Dist: flask; extra == "examples"
Requires-Dist: streamlit; extra == "examples"
Requires-Dist: jupyterlab; extra == "examples"
Provides-Extra: winlid
Provides-Extra: corpus
Requires-Dist: datasets>=3.0; extra == "corpus"
Requires-Dist: pyarrow>=14.0; extra == "corpus"
Requires-Dist: requests>=2.0; extra == "corpus"
Provides-Extra: sentry
Requires-Dist: sentry-sdk>=2.0; extra == "sentry"
Provides-Extra: dev
Requires-Dist: ruff>=0.2.0; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=4.0; extra == "dev"
Requires-Dist: make>=0.1.6; extra == "dev"
Requires-Dist: types-requests>=2.31; extra == "dev"
Dynamic: license-file

---
title: Turkic Transliteration Demo
emoji: 🌖
colorFrom: green
colorTo: green
sdk: gradio
sdk_version: 5.29.0
app_file: app.py
pinned: false
license: apache-2.0
short_description: Transliteration of Turkic languages into Latin and IPA
---

# turkic_transliterate

Deterministic Latin and IPA transliteration for Turkic languages, plus helper
utilities for corpus building, tokenizer/LM training, and Russian-token
filtering. Languages are discovered dynamically from rule files in
`src/turkic_translit/rules/`, so support grows by adding a `.rules` file.

**Supported languages** (verified by the test suite): Azerbaijani, Finnish,
Kazakh, Kyrgyz, Turkish, Uyghur, and Uzbek (both Cyrillic and Latin input).
IPA output is available for all of them; Latin output covers Kazakh and Kyrgyz
(plus embedded Arabic-script handling). The batch `turkic-translit translit`
CLI, the library API (`to_ipa`/`to_latin`), and the web demo all cover the
full set — language choices are discovered dynamically from the rules
directory.

## Install

This project uses **Poetry**. Python **3.9+**.

```bash
# Simplest: the dev setup script (also installs the Windows PyICU wheel)
python scripts/setup_dev.py

# Or with Poetry directly
poetry install

# Or with pip (editable, with dev + web-UI extras)
pip install -e .[dev,ui]        # add ,winlid on Windows for the fasttext wheel
```

Optional extras: `dev` (ruff, mypy, pytest) · `ui` (Gradio demo) ·
`winlid` (Windows fasttext wheel for language ID) · `sentry` (error reporting).

### Windows & PyICU

PyPI rules prevent the correct Windows PyICU wheel from installing automatically
during `pip install`. After installing, Windows users run:

```bash
turkic-pyicu-install
```

which fetches the right PyICU wheel for your Python version. `scripts/setup_dev.py`
does this for you.

### Package names

- Import path: `turkic_translit`
- PyPI distributable: `turkic_transliterate`
- Primary CLI entry point: `turkic-translit`

## Development

The Makefile wraps the common tasks (all via Poetry):

```bash
make check   # lint + test (the full gate)
make lint    # ruff check --fix, ruff format, mypy --strict
make test    # pytest
make web     # launch the Gradio web UI
make help    # list all targets
```

On Windows without GNU Make, use the PowerShell wrapper:

```powershell
./scripts/run.ps1 lint
./scripts/run.ps1 test
./scripts/run.ps1 web
./scripts/run.ps1 help
```

Type-checking is `mypy --strict` (config in `mypy.ini`, scoped to `src/`).
Formatting and linting are both handled by **ruff** (`ruff format` replaced
black).

## Command-line tools

Console entry points defined in `pyproject.toml`:

| Command | Purpose |
|---------|---------|
| `turkic-translit` | Transliterate text to Latin and/or IPA (`--lang kk\|ky`) |
| `turkic-filter-russian` | Drop or mask Russian tokens from a stream |
| `turkic-download-corpus` | Download/prepare OSCAR corpora |
| `turkic-build-spm` / `turkic-train-spm` | Train a SentencePiece tokenizer |
| `turkic-train-lm` / `turkic-eval-lm` | Train / evaluate a language model |
| `turkic-leven` | Levenshtein-based comparison utility |
| `turkic-web` | Launch the Gradio web demo |
| `turkic-pyicu-install` | Install the correct PyICU wheel (Windows) |

### `turkic-translit translit` usage

```bash
turkic-translit translit --lang kk --in text.txt \
    --out-latin kk_lat.txt --out-ipa kk_ipa.txt --arabic
```

- `--lang` — any ISO 639-1 code advertised by the rules directory
  (`az`, `fi`, `kk`, `ky`, `tr`, `ug`, `uz`)
- `--in` — input file, or `-` for stdin (default: `-`)
- `--out-latin` — Latin output path, or `-` for stdout. Omit to skip.
  Only meaningful for languages with `<lang>_lat.rules` (`kk`, `ky`).
- `--out-ipa` — IPA output path, or `-` for stdout. Omit to skip.
- `--arabic` — also transliterate embedded Arabic script (Latin mode only)
- `--benchmark` — log throughput statistics on completion

At least one of `--out-latin` / `--out-ipa` must be specified.
`--log-level` is set on the parent group: `turkic-translit --log-level debug translit ...`.

### Tokenizer training

```bash
turkic-build-spm --input corpora/kk_lat.txt,corpora/ky_lat.txt \
    --model_prefix spm/turkic12k --vocab_size 12000
```

### Filtering Russian tokens

```bash
cat uz_raw.txt | turkic-filter-russian --mode drop > uz_clean.txt
```

## Demos via `turkic_tools.py`

```bash
python turkic_tools.py [command]
```

- `web` — launch the Gradio web interface
- `demo` — simple CLI demo
- `full-demo` — comprehensive multi-language demo
- `help` — list commands

## Project Organization

- `src/turkic_translit/` — core package (`core.py`, `transliterate.py`,
  `rules/`, `cli/`, `web/`, `lm/`, language-ID and filtering modules)
- `data/` — sample data and language resources
- `docs/` — documentation and setup/troubleshooting guides
- `scripts/` — dev + release utilities (`setup_dev.py`, `run.ps1`, `release/`)
- `vendor/` — pre-built PyICU wheels for Windows
- `tests/` — test suite (per-language IPA coverage for all supported languages)
- `cronjob/` — scheduled-task assets for the hosted demo
- `app.py` — Hugging Face Space entry point for the web UI

## FastText language-identification model

Russian-token filtering and language detection use FastText's `lid.176.bin`.
The model is **not** committed (too large); it is downloaded automatically from
the official Facebook AI link on first use and cached in the package directory —
no manual step needed for pip installs, Hugging Face Spaces, or CI. Manual
source if needed:
https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin

## Logging & error reporting

Central logging supports structured JSON with correlation IDs. Control with
`TURKIC_LOG_LEVEL` (DEBUG…ERROR) and `TURKIC_LOG_FORMAT=json|rich` (default
json); libraries can call `turkic_translit.logging_config.setup()`. Optional
Sentry integration via `TURKIC_SENTRY_DSN` (+ `TURKIC_ENV`,
`TURKIC_SENTRY_TRACES`); install with `pip install turkic-translit[sentry]`.

All I/O is UTF-8; on Windows a BOM is written when piping to files to avoid
encoding issues.

## Relationship to the LSTM project

This repo is the upstream data pipeline for the Turkic mutual-intelligibility
LSTM experiments (github.com/wagner-austin/LSTM): `turkic-download-corpus` plus
the IPA transliteration rules produce the per-language IPA corpora that project
trains on.

## License

Apache-2.0
