Metadata-Version: 2.5
Name: orthomni-codeco
Version: 0.1.2
Summary: Vietnamese spelling correction (BARTpho seq2seq) on CPU with CTranslate2 / ONNX Runtime INT8 - no PyTorch needed
Project-URL: Homepage, https://huggingface.co/nhatho1999/orthomni-codeco
Author-email: "Nhat H. Pham" <nhatho1999.ai@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: bartpho,ctranslate2,nlp,onnx,onnxruntime,spelling-correction,vietnamese
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Vietnamese
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: ctranslate2>=4.0
Requires-Dist: huggingface-hub>=0.20
Requires-Dist: numpy>=1.21
Requires-Dist: sentencepiece>=0.1.99
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Provides-Extra: export
Requires-Dist: ctranslate2>=4.0; extra == 'export'
Requires-Dist: onnx>=1.15; extra == 'export'
Requires-Dist: optimum-onnx[onnxruntime]>=0.1.0; extra == 'export'
Requires-Dist: torch>=2.1; extra == 'export'
Requires-Dist: transformers<4.58,>=4.45; extra == 'export'
Provides-Extra: onnx
Requires-Dist: onnxruntime>=1.16; extra == 'onnx'
Provides-Extra: server
Requires-Dist: fastapi>=0.110; extra == 'server'
Requires-Dist: pydantic>=2.6; extra == 'server'
Requires-Dist: uvicorn[standard]>=0.29; extra == 'server'
Description-Content-Type: text/markdown

# orthomni-codeco

Vietnamese spelling correction on **CPU** with **INT8** inference, **no PyTorch required**. The model is a Seq2Seq BARTpho/MBart (~400M parameters).

- **Fast by default:** runs on [CTranslate2](https://github.com/OpenNMT/CTranslate2), which keeps the encoder, the decoding loop and the KV-cache in native C++ code. [ONNX Runtime](https://onnxruntime.ai/) is available as an alternative backend.
- **Lightweight:** the only dependencies are `ctranslate2`, `numpy`, `sentencepiece` and `huggingface-hub` (versus ~1.3 GB with torch + transformers).
- **Automatic model download:** the wheel contains no weights. On first run, only the files of the selected backend (~400 MB) are downloaded from the Hugging Face Hub to `~/.cache/orthomni_codeco/`, with a progress bar. Later runs work offline.
- **Batteries included:** FastAPI server, CLI and Dockerfile.

## Installation

```bash
pip install orthomni-codeco            # library (CTranslate2 backend)
pip install "orthomni-codeco[server]"  # + FastAPI/uvicorn
pip install "orthomni-codeco[onnx]"    # + ONNX Runtime backend (optional)
```

Requires Python >= 3.9.

## Quickstart

```python
from orthomni_codeco import OrthoMNIModel

model = OrthoMNIModel()                         # first run: downloads the model to the cache, then loads it (<1 s)
model.correct("hom nay troi dep qua")           # 'Hôm nay trời đẹp quá.'
model.correct(["xin chao", "toi di hoc"])       # ['Xin chào.', 'Tôi đi học.']
```

`predict()` and `generate()` are aliases of `correct()`. `Corrector` is the old name of `OrthoMNIModel`.

## Library usage

```python
from orthomni_codeco import OrthoMNIModel, CorrectorConfig, InputTooLongError, get_corrector, ensure_model

model = get_corrector()                                   # per-process singleton, config read from ORTHOMNI_* env vars
model = OrthoMNIModel(CorrectorConfig(intra_op_threads=4))
model = OrthoMNIModel(CorrectorConfig(model_dir="/path/to/int8"))  # use a local model, no download

await model.acorrect("xin chao")                          # async, does not block the event loop

try:
    model.correct(very_long_text)
except InputTooLongError as e:
    print(e.indices, e.max_tokens)                        # which inputs exceed the limit -> split sentences first

ensure_model()                                            # only download the model to the cache, returns its path
```

- **Load once:** create `OrthoMNIModel` once and reuse it.
- **Thread-safe:** calls are serialized with a lock, since the backend already uses all the threads it is given.
- **Lightweight import:** `import orthomni_codeco` does not load ctranslate2, onnxruntime or numpy.

### Backends

| Backend | Install | Model files | Notes |
|---|---|---|---|
| `ct2` (default) | included | `ct2/model.bin` (~400 MB) | CTranslate2 INT8; encoder, greedy decoding and KV-cache all run in C++ |
| `onnx` | `pip install "orthomni-codeco[onnx]"` | `onnx/*.onnx` (~470 MB) | ONNX Runtime INT8; decoding loop in Python |

`backend="auto"` (default) picks `ct2` when `ctranslate2` is installed, otherwise `onnx`. Choose explicitly with `CorrectorConfig(backend="onnx")`, `ORTHOMNI_BACKEND=onnx` or `--backend onnx` on the CLI. Both backends use the same tokenizer. The current model on the Hub (package >= 0.1.2) ships only `ct2/`; the `onnx` backend needs a local model directory exported with `--backends onnx`.

### Model source

In order of priority:
1. `model_dir` / `ORTHOMNI_MODEL_DIR`: a local model directory, nothing is downloaded.
2. `repo_id` + `revision` (`ORTHOMNI_REPO_ID`, `ORTHOMNI_REVISION`): downloaded from the Hugging Face Hub to `cache_dir` (`ORTHOMNI_CACHE_DIR`, default `~/.cache/orthomni_codeco/<org>--<repo>/<revision>/`).

Private repos use `HF_TOKEN` or `hf auth login`. For offline environments, pre-download with `orthomni download`, then set `HF_HUB_OFFLINE=1`.

## Mounting into an existing FastAPI app

```python
import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI
from orthomni_codeco import get_corrector
from orthomni_codeco.server import create_router

@asynccontextmanager
async def lifespan(app: FastAPI):
    await asyncio.to_thread(get_corrector)   # load the model at startup
    yield

app = FastAPI(lifespan=lifespan)
app.include_router(create_router(), prefix="/spell")   # -> POST /spell/predict, GET /spell/health
```

## CLI and standalone server

```bash
orthomni download                                   # download the model to the cache, print its path (--backend onnx for ONNX)
orthomni correct "toi di hoc" "xin chao"            # quick test
orthomni serve --port 8006                          # workers/threads derived from the vCPU count
orthomni serve --workers 1 --threads 8              # lowest latency
```

`python -m orthomni_codeco ...` is equivalent to `orthomni ...`.

| Method | Path | Description |
|---|---|---|
| GET | `/health` | `200 {"status":"ok"}` when the model is ready, `503` otherwise |
| POST | `/predict` | Body `{"text": str}` **or** `{"texts": list[str]}` |

```bash
curl -X POST localhost:8006/predict -H 'content-type: application/json' -d '{"text":"hom nay troi dep qua"}'
# {"result":"Hôm nay trời đẹp quá.","latency_ms":113.7}
```

Error codes: `422` invalid input (both `text` and `texts`, batch too large, empty text), `413` sentence longer than `MAX_INPUT_TOKENS` (the server does not truncate), `500` inference error.

## Configuration (environment variables)

| Variable | Default | Meaning |
|---|---|---|
| `ORTHOMNI_MODEL_DIR` | (empty) | Local model directory; when set, no download happens |
| `ORTHOMNI_REPO_ID` / `ORTHOMNI_REVISION` | see `config.py` | Hugging Face Hub repo and revision |
| `ORTHOMNI_CACHE_DIR` | `~/.cache/orthomni_codeco` | Model cache directory |
| `ORTHOMNI_BACKEND` | `auto` | `auto`, `ct2` or `onnx` |
| `ORTHOMNI_INTRA_OP_THREADS` | `0` (= vCPU / workers) | Compute threads within one operator (both backends). **Biggest effect on latency** |
| `ORTHOMNI_INTER_OP_THREADS` | `1` | (onnx) Parallelism across graph branches |
| `ORTHOMNI_COMPUTE_TYPE` | `int8` | (ct2) CTranslate2 compute type |
| `WEB_CONCURRENCY` | `vCPU / 4` | Number of uvicorn workers; each holds its own model copy (~1.7 GB RAM) |
| `ORTHOMNI_MAX_INPUT_TOKENS` | `256` | Maximum tokens per sentence |
| `ORTHOMNI_INFER_BATCH_SIZE` | `8` | Sub-batch size (sorted by length to reduce padding) |
| `ORTHOMNI_NEW_TOKENS_RATIO` / `_OFFSET` / `_CAP` | `1.3` / `8` / `320` | `max_new_tokens = min(CAP, len_input × RATIO + OFFSET)` |
| `ORTHOMNI_WARMUP` | `true` | Run one dummy inference on load |
| `ORTHOMNI_MAX_BATCH_SIZE` | `32` | Maximum number of `texts` per request |
| `ORTHOMNI_MAX_TEXT_CHARS` | `4000` | Maximum characters per text |
| `ORTHOMNI_MAX_CONCURRENCY` | `1` | Concurrent generate requests per worker |

**Avoid thread thrashing:** keep `workers × ORTHOMNI_INTRA_OP_THREADS ≤ vCPUs`. The `serve` command reads CPU limits from cgroups (`docker --cpus`, K8s limits). Fewer workers with more threads gives lower latency; more workers with fewer threads gives higher throughput.

## Docker

Requires the INT8 model in `./model`, laid out like the Hugging Face repo (`ct2/`, `onnx/`, tokenizer files; see below):

```bash
docker build -t orthomni-codeco:cpu .
docker run --rm -p 8006:8006 --cpus 8 --memory 4g orthomni-codeco:cpu
```

## Development

### Environment setup

```bash
python -m venv .venv && . .venv/bin/activate
pip install -e ".[server,dev]"
ORTHOMNI_MODEL_DIR=./model pytest -q        # model tests are skipped if ./model is missing
./run.sh                                    # local server using ./model, port 7428
```

### Exporting a new model

```bash
pip install torch --index-url https://download.pytorch.org/whl/cpu   # CPU-only torch first
pip install -e ".[export]"
orthomni export --model_dir /path/to/checkpoint --output_dir ./model [--backends ct2 onnx] [--target avx512_vnni]
```

- `ct2/`: CTranslate2 INT8 (all linear layers and embeddings). Independent of the CPU architecture.
- `onnx/`: dynamic INT8 **per-channel** on `MatMul` (per-tensor quantization produced incorrect sentences), then compacted: embeddings become INT8 per-row, and the cross-attention K/V projection moves into the encoder so `decoder_model.onnx` (a ~90% duplicate of `decoder_with_past_model.onnx`) is dropped. ONNX INT8 kernels depend on the CPU, so export on a machine with the same architecture as the serving machine (`--target`).

To convert a 0.1.0 model directory (3 ONNX files) without the original checkpoint, use `python scripts/convert_v010_model.py --src ./old_model --dst ./model`. The script rebuilds an FP32 checkpoint from the INT8 weights; since they already sit on the symmetric INT8 grid, CTranslate2 re-quantizes them to the same values.

### Publishing the model to the Hugging Face Hub

```bash
hf auth login
python scripts/upload_model.py --model_dir ./model --repo_id <org>/orthomni-codeco
```

The script prints a commit hash. Update `DEFAULT_REPO_ID` and `DEFAULT_REVISION` in `src/orthomni_codeco/config.py` before releasing, so each package version always downloads the same set of weights.

### Build and publish

```bash
pip install -U build twine
rm -rf dist/ && python -m build                  # -> dist/*.whl + dist/*.tar.gz
twine check dist/*                               # check metadata and that the README renders on PyPI
unzip -l dist/*.whl                              # confirm no model files (.onnx/.bin) are inside

# Try the wheel in a clean venv
python -m venv /tmp/t && /tmp/t/bin/pip install dist/*.whl
/tmp/t/bin/python -c "from orthomni_codeco import OrthoMNIModel; print(OrthoMNIModel().correct('xin chao'))"

twine upload --repository testpypi dist/*        # try on TestPyPI first
twine upload dist/*                              # real PyPI
```

## License

MIT
