Metadata-Version: 2.5
Name: ogonki
Version: 0.1.0
Summary: Restore Polish diacritics (zolw zjadl salate -> żółw zjadł sałatę). Tiny, offline, numpy-only.
Project-URL: Homepage, https://github.com/kjanus03/ogonki
Project-URL: Issues, https://github.com/kjanus03/ogonki/issues
Author: Kacper Janus
License-Expression: MIT
License-File: LICENSE
Keywords: diacritics,diacritics-restoration,nlp,ogonki,polish,polski
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Polish
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: numpy
Provides-Extra: train
Requires-Dist: datasets; extra == 'train'
Requires-Dist: torch; extra == 'train'
Description-Content-Type: text/markdown

# ogonki

[![PyPI](https://img.shields.io/pypi/v/ogonki)](https://pypi.org/project/ogonki/)
[![Python](https://img.shields.io/pypi/pyversions/ogonki)](https://pypi.org/project/ogonki/)
[![CI](https://github.com/kjanus03/ogonki/actions/workflows/publish.yml/badge.svg)](https://github.com/kjanus03/ogonki/actions)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/kjanus03/ogonki/blob/main/LICENSE)

Restore Polish diacritics in text typed without them. A tiny character-level model that runs offline with numpy only.

![demo](https://raw.githubusercontent.com/kjanus03/ogonki/main/demo.gif)

```bash
pip install ogonki
```

```python
>>> from ogonki import restore
>>> restore("Wczoraj bylem w Lodzi")
'Wczoraj byłem w Łodzi'
>>> restore("Zolw zjadl salate")
'Żółw zjadł sałatę'
>>> df["title"] = df["title"].map(restore)   # pandas
```

```bash
ogonki notes.txt > fixed.txt
echo "Sprzedam rower gorski, stan bardzo dobry, odbior osobisty w Gdansku" | ogonki
```

- **Tiny and offline:** a 0.8 MB model, with numpy as the only dependency. No GPU, API key or download on first use.
- **Fast:** about 125,000 characters per second on a laptop CPU.
- **Safe:** only `a c e l n o s z` can change, and the output always has the same length as the input. Words that already have Polish letters, URLs and e-mail addresses are left untouched.

## Benchmark

Word accuracy on 300,000 characters of held-out Polish web text ([FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2)), counting only words that could take a diacritic. Reproduce with `python train.py`.

| method | word accuracy |
|---|---|
| no change | 63.34% |
| dictionary (most frequent spelling seen in training) | 96.80% |
| **ogonki** | **98.46%** |

ogonki makes about half as many mistakes as a dictionary lookup (1.5% of words instead of 3.2%).

## Limitations

- **Rare words** can be missed, for example in the pangram *Zażółć gęślą jaźń*.
- **Words whose spelling depends on meaning** usually get the more common form: *sad* (orchard) vs *sąd* (court), *czuje* (he feels) vs *czuję* (I feel), *laska* (cane) vs *łaska* (grace).
- The model was trained and tested on web text only. Accuracy on OCR output, chat messages or old documents hasn't been measured yet.

## How it works

For every `a c e l n o s z`, a small character-level CNN (8 dilated convolution layers that see 38 characters on each side) picks one of three classes: *leave it*, *add the usual diacritic* (a→ą, l→ł, z→ż…), or *z→ź*. The training data is free: take Polish text, strip its diacritics, and learn to put them back.

```
ogonki/__init__.py   restore(), strip(), CLI    (~75 lines, numpy only)
ogonki/weights.npz   trained model              (produced by train.py)
train.py             data -> train -> export -> benchmark
```

## Train it yourself

Training takes under an hour on a free Google Colab T4 GPU, and much longer on a CPU.

```bash
pip install -e ".[train]"
python train.py
```

## Data

Trained on the Polish part of [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) by Hugging Face, released under the ODC-By 1.0 license.

## License

MIT
