Metadata-Version: 2.5
Name: bhashantar
Version: 0.1.0
Summary: IndicTrans2 English-to-Indic translation on CPU via CTranslate2 int8, no transformers or fairseq at runtime
Author-email: Zaid Ali Syed <sidzaid72@gmail.com>
License-Expression: MIT
License-File: LICENSE
License-File: NOTICE
Keywords: ctranslate2,hindi,indic,indictrans2,machine-translation
Classifier: Natural Language :: Hindi
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: ctranslate2<5,>=4.4
Requires-Dist: huggingface-hub>=0.24
Requires-Dist: regex>=2023.0
Requires-Dist: sacremoses>=0.1.1
Requires-Dist: sentencepiece<0.3,>=0.2
Provides-Extra: convert
Requires-Dist: safetensors>=0.4; extra == 'convert'
Provides-Extra: dev
Requires-Dist: psutil>=5.9; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: sacrebleu>=2.4; extra == 'dev'
Description-Content-Type: text/markdown

# bhashantar

English to 25 Indic languages with AI4Bharat's IndicTrans2 (`en-indic-dist-200M`), on CPU, through CTranslate2 int8. No transformers, no torch, no fairseq and no IndicTransToolkit at runtime.

```python
from bhashantar import translate

translate(["The weather in Mumbai is pleasant today."], tgt="hin_Deva")
# ['मुंबई में आज मौसम सुखद है।']
```

```
$ bhashantar translate --tgt hin_Deva "Send the report to ops@example.com by 15-03-2025."
रिपोर्ट को 15-03-2025 द्वारा ops@example.com पर भेजें।
```

## Why this exists

The official checkpoint loads through `trust_remote_code` files written against transformers 4.3x. On a fresh Python 3.12 venv with transformers 5.22.2 and IndicTransToolkit 1.1.1 (reproduced 2026-11-09):

```
from IndicTransToolkit.processor import IndicProcessor
ImportError: cannot import name 'PreTrainedTokenizerBase' from 'transformers.tokenization_utils'

AutoTokenizer.from_pretrained(..., trust_remote_code=True)
ModuleNotFoundError: No module named 'transformers.onnx'

AutoModelForSeq2SeqLM.from_pretrained(..., trust_remote_code=True)
ModuleNotFoundError: No module named 'transformers.onnx'
```

This is [IndicTransToolkit#29](https://github.com/VarunGumma/IndicTransToolkit/issues/29). A fix for the remote code is open as discussion #15 on the model repo, unmerged at the time of writing. Pinning transformers 4.50.3 makes the stack load again; that pinned stack is the reference this package is measured against.

## Data path

```
raw text
  -> punctuation and digit normalisation, URLs/emails/numbers swapped for <IDn>   (textprep.prepare)
  -> Moses tokenisation, "eng_Latn hin_Deva" tags, SentencePiece pieces           (engine)
  -> CTranslate2 int8 beam search (beam 5), Devanagari-unified pieces out
  -> detokenise pieces, restore <IDn>, Devanagari -> target script, reattach punctuation   (textprep.finish)
```

The model writes every Brahmic target in Devanagari; Tamil, Bengali and the rest come from a fixed Unicode offset afterwards. That step and the placeholder restore are where a port silently goes wrong, so both are vendored from IndicTransToolkit and indic_nlp_library (MIT, see `NOTICE`) with their regexes and quirks intact. Two deliberate changes: placeholder maps travel with each sentence instead of a shared queue, and placeholder numbering is sorted instead of following set order, which varied between processes.

## Two ways this fails, and the guard for each

**Silent quality loss from conversion or quantisation.** CTranslate2 has no converter for this checkpoint's custom class, so `bhashantar.convert` maps the Hugging Face tensors onto the same `TransformerSpec` the fairseq converter fills. A wrong position offset or a missed layer norm still produces fluent-looking text. The guard is `bench/parity.py`: chrF against human references for both this package and the upstream model, with the pass bar fixed before the first run.

**Breakage on library upgrades.** Runtime dependencies are only what inference touches: `ctranslate2` (capped below 5, since `model.bin` is tied to the major), `sentencepiece`, `sacremoses`, `regex`, `huggingface_hub`. `.github/workflows/smoke.yml` installs the newest versions in range every week and translates.

**The opinion:** parity is published as a number, not claimed. If the delta ever exceeds the bar, the table says so.

## Parity

FLORES-200 devtest, first 100 sentences, English source. The FLORES files come from the public `dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz`; AI4Bharat's IN22 and the Hugging Face FLORES+ copy are both gated. Reference = upstream model, transformers 4.50.3 + IndicTransToolkit 1.1.1, fp32, beam 5. Pass bar, set before running: |delta| <= 1.0 chrF.

| pair | chrF reference | chrF bhashantar | delta | identical outputs | pass |
|---|---|---|---|---|---|
| eng_Latn-hin_Deva | 65.69 | 65.17 | -0.52 | 61/100 | yes |
| eng_Latn-ben_Beng | 54.52 | 54.62 | +0.10 | 54/100 | yes |
| eng_Latn-tam_Taml | 61.33 | 61.59 | +0.27 | 56/100 | yes |

Reproduce: `python bench/parity.py score <flores devtest dir> bench/results` (outputs and `parity.json` are committed in `bench/results`).

"Identical outputs" counts sentences where both systems produce the same string; the rest differ in word choice (for example कनाडाई मधुमेह संघ against कैनेडियन डायबिटीज एसोसिएशन), the expected effect of int8 weights on beam search. Conversion error and quantisation error are not separated: an fp32 CTranslate2 conversion did not fit on the build machine's disk.

Reference weights: the official repo is gated, so the reference used `Raghavan/indictrans2-en-indic-dist-200M`, whose `pytorch_model.bin` has the same Git LFS pointer as the official file (sha256 `52671a26...0fef0`, verified locally); tokenizer files match the official Git blob ids.

## Speed and memory

| | bhashantar (CT2 int8) | upstream (transformers fp32) |
|---|---|---|
| model load | 0.47 s | not timed |
| ms per sentence, eng to tam_Taml, beam 5 | 323 (all 100 in one call) | 2,786 (batch 2, 80 sentences timed) |
| peak RSS | 707 MB | 1,472 MB |

The two latency figures are not like for like: the reference had to run at batch 2 because the build machine ran out of commit memory at batch 8, and it was resumed once mid-run, so only the last 80 sentences were timed. bhashantar on Hindi and Bengali: 328 and 308 ms per sentence.

Measured on a laptop (Windows 11, 16 GB RAM, 4 threads, other jobs running), so treat them as relative, not absolute.

## Limits

- One direction: English to Indic. The `indic-en` and `indic-indic` checkpoints are not converted.
- Caps, refused with the cap named in the error: 1,000 characters per sentence, 512 sentences and 50,000 characters per call, 256 subword tokens per sentence after preprocessing. Long input is refused, not truncated.
- Santali (`sat_Olck`) output mixes in Oriya and Arabic words for some sentences. The upstream model does the same; the test marks it as a known model limit.
- Sentence-level: split paragraphs yourself.

## Model files

`translate()` looks for a model in this order: the `model_dir` argument, `$BHASHANTAR_MODEL`, a local conversion at `$HF_HOME/bhashantar/en-indic-dist-200M-ct2-int8`, then the Hugging Face repo `zaidwhy/indictrans2-en-indic-dist-200M-ct2-int8` (cache first, so it works offline after the first download).

That repo is not published yet. Until it is, convert once yourself (needs a Hugging Face login with the model's terms accepted):

```
pip install "bhashantar[convert]"
hf auth login
bhashantar convert
```

## Development

```
pip install -e ".[dev]"
pytest                 # offline tests, no model
pytest -m model        # end to end, needs the converted model
```

## Maintainer: publishing (not done yet)

```
# 1. weights, from a machine with the converted model
hf auth login
hf repo create zaidwhy/indictrans2-en-indic-dist-200M-ct2-int8 --repo-type model
hf upload zaidwhy/indictrans2-en-indic-dist-200M-ct2-int8 ~/.cache/huggingface/bhashantar/en-indic-dist-200M-ct2-int8 .

# 2. check that a clean install fetches it
python -m venv /tmp/bt && /tmp/bt/bin/pip install . && /tmp/bt/bin/bhashantar translate --tgt tam_Taml "Good morning."

# 3. package
python -m pip install build twine
python -m build
twine check dist/*
twine upload dist/*
```

The model directory carries AI4Bharat's MIT `LICENSE` and a `provenance.json` with the source weights' sha256; add a model card stating the parity numbers above before uploading.

## License

MIT. Vendored code and model weights keep their own MIT notices; see `NOTICE`.
