Metadata-Version: 2.4
Name: khocr-gen
Version: 0.1.0
Summary: Synthetic OCR training data generator for mixed Khmer/English text
Project-URL: Homepage, https://github.com/LazyGreed/khocr-gen
Project-URL: Repository, https://github.com/LazyGreed/khocr-gen
Project-URL: Issues, https://github.com/LazyGreed/khocr-gen/issues
Project-URL: Changelog, https://github.com/LazyGreed/khocr-gen/blob/main/CHANGELOG.md
Author: Kosal Seng
License-Expression: MIT
License-File: LICENSE
Keywords: data-generation,deep-learning,image-generation,khmer,ocr,synthetic-data,text-recognition
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Text Processing :: Fonts
Requires-Python: >=3.12
Requires-Dist: albumentations>=1.3.0
Requires-Dist: khmernormalizer>=0.0.4
Requires-Dist: lmdb>=1.4.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: opencv-python-headless>=4.8.0
Requires-Dist: pillow>=10.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: tqdm>=4.64.0
Provides-Extra: dev
Requires-Dist: khocr-gen-core; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: rust
Requires-Dist: khocr-gen-core; extra == 'rust'
Description-Content-Type: text/markdown

# khocr-gen

Synthetic OCR training data generator for mixed Khmer/English text.

## Features

- **Mixed-script rendering:** per-span font selection for Khmer + English
- **24 augmentation methods:** unified registry covering scanner/camera degradations and training-time transforms
- **Rust acceleration:** 21/24 augmentation methods plus font glyph checking run through a native PyO3 extension, with automatic pure-Python fallback
- **Isolated augmentation:** one effect per image, weighted by configurable probabilities
- **Configurable intensity ranges:** per-method `[min, max]` with linear sampling
- **Multiprocess generation:** parallel workers for throughput
- **LMDB packing:** efficient key–value storage for training pipelines
- **n-way dataset combine:** merge multiple generated datasets into one
- **Built-in viewer:** preview and extract images from LMDB databases
- **YAML config files:** reproducible generation recipes
- **Khmer normalization:** canonical character ordering via `khmernormalizer`

## Installation

```bash
# Clone and install with uv
git clone https://github.com/LazyGreed/khocr-gen.git
cd khocr-gen
uv sync

# Or install with pip
pip install .
```

`uv sync` also compiles and installs the optional Rust acceleration extension
(`khocr-gen-core`, from `rust/`). This requires a Rust toolchain (`cargo`/`rustc`) on
`PATH` — see [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md). If the extension isn't
available, khocr-gen falls back to pure-Python implementations automatically.

### Requirements

- Python >= 3.12
- Rust toolchain (`cargo`/`rustc`, edition 2021) — only needed to build the optional
  acceleration extension; the project works without it

## Quick Start

### 1. Prepare fonts

```text
fonts/
├── khmer/     <- .ttf / .otf Khmer fonts
└── english/   <- .ttf / .otf English fonts
```

Fonts placed directly in `fonts/` (not in a subdirectory) are added to both pools as fallbacks.

### 2. Prepare corpus

A plain UTF-8 text file, one string per line:

```text
សួស្តី
Hello World
ស្វាគមន៍ Welcome
```

### 3. Generate a dataset

```bash
khocr-gen generate --corpus corpus/corpus.txt --fonts fonts --output data --copies 3 --storage lmdb
```

### 4. Verify augmentation quality

Renders every augmentation method at min and max intensity on clean canvases:

```bash
khocr-gen verify --fonts fonts --output verify_output
```

### 5. View generated images

```bash
# Summary of an LMDB database
khocr-gen view --lmdb data/train/lmdb

# Extract images
khocr-gen view --lmdb data/train/lmdb --output-dir extracted

# Show labels only
khocr-gen view --lmdb data/train/lmdb --labels-only
```

### 6. Combine multiple datasets

```bash
khocr-gen combine data_run1 data_run2 data_run3 --output data_merged
```

## Configuration

All generation parameters can be specified via CLI flags, a YAML config file, or both (CLI overrides YAML).
See [CONFIG.md](docs/CONFIG.md) for details.

```bash
khocr-gen generate --config configs/generate.yml
```

Config loading order (highest priority wins):
1. argparse built-in defaults
2. YAML config file values
3. Explicit CLI flags

## Augmentation

24 methods in a unified registry (21 of them Rust-accelerated).
Each generated image receives exactly one augmentation, chosen probabilistically by weight.
See [AUGMENTATION.md](docs/AUGMENTATION.md) for the full catalog and visual examples.

## CLI Reference

See [CLI_REFERENCE.md](docs/CLI_REFERENCE.md) for complete command documentation.

## Rust Acceleration

See [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md) for what's accelerated, how the
native extension is built/installed, and how to iterate on the `rust/` crate.

## Project Structure

```text
src/khocr_gen/
├── cli.py              # CLI entry point
├── config.py           # GenerationConfig + AugMethodConfig dataclasses
├── config_loader.py    # YAML config loading
├── corpus.py           # Corpus loading / filtering / counting
├── fonts.py            # FontManager with script-aware font selection
├── normalizer.py       # Khmer text normalization wrapper
├── rendering.py        # ImageRenderer: text -> clean canvas -> augmentation
├── augmentation.py     # 24 augmentation methods (unified registry)
├── _rust_accel.py      # Native/pure-Python dispatch layer for the Rust extension
├── parallel.py         # Multiprocess rendering workers
├── lmdb_pack.py        # LMDB database packing
├── combine.py          # n-way dataset merge
├── data_generator.py   # DatasetGenerator orchestrator
├── generate.py         # generate CLI command
├── verify.py           # verify CLI command
├── viewer.py           # view CLI command
├── combine_cmd.py      # combine CLI command
└── errors.py           # Exception hierarchy

rust/                    # khocr-gen-core: PyO3/maturin native acceleration extension
├── Cargo.toml
├── pyproject.toml
└── src/
    ├── lib.rs           # PyO3 bindings + module registration
    ├── augmentation.rs  # Accelerated augmentation methods
    ├── fonts.rs         # FontManager + FontFace (cmap glyph lookup)
    ├── rendering.rs     # Unwired canvas-rendering prototype (not yet bound to Python)
    └── utils.rs
```

## Acknowledgments

khocr-gen builds on excellent open-source libraries:

- [khmernormalizer](https://github.com/seanghay/khmernormalizer) (MIT) - Khmer text normalization
- [OpenCV](https://opencv.org/) (Apache 2.0) - Image processing and augmentation
- [Pillow](https://python-pillow.org/) (HPND) - Font rendering and image creation
- [Albumentations](https://albumentations.ai/) (MIT) - Augmentation primitives
- [NumPy](https://numpy.org/) (BSD-3-Clause) - Numerical operations
- [LMDB](https://www.symas.com/lmdb) (OpenLDAP 2.8) - Embedded database for training pipelines

## License

MIT
