Metadata-Version: 2.4
Name: indicate
Version: 0.10.0
Summary: Transliterations to/from Indian languages
Keywords: transliterate,indic,hindi,english
Author: Rajashekar Chintalapati, Gaurav Sood
Author-email: Rajashekar Chintalapati <rajshekar.ch@gmail.com>, Gaurav Sood <gsood07@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Requires-Dist: click>=8.0.0
Requires-Dist: tqdm>=4.60.0
Requires-Dist: numpy>=1.26
Requires-Dist: torch>=2.6.0
Requires-Dist: safetensors>=0.4.0
Requires-Dist: huggingface-hub>=0.23
Requires-Dist: litellm>=1.0.0
Requires-Dist: batchlane>=0.1.0
Requires-Python: >=3.13
Project-URL: Homepage, https://github.com/in-rolls/indicate
Project-URL: Repository, https://github.com/in-rolls/indicate
Description-Content-Type: text/markdown

# Indicate: Transliterate Indic Languages with PyTorch and LLMs

[![PyPI Version](https://img.shields.io/pypi/v/indicate.svg)](https://pypi.python.org/pypi/indicate)
[![Downloads](https://static.pepy.tech/badge/indicate)](https://pepy.tech/project/indicate)
[![Tests](https://github.com/in-rolls/indicate/workflows/CI/badge.svg)](https://github.com/in-rolls/indicate/actions?query=workflow%3ACI)
[![Documentation](https://img.shields.io/badge/docs-github.io-blue)](https://in-rolls.github.io/indicate/)

**Indicate** provides high-quality transliteration between Indic languages and English using both a traditional PyTorch model and state-of-the-art LLMs (Large Language Models).

## 🚀 Features

- **🔀 Composable Backends**: Chain a word table, a local model and an LLM in any order
- **🌍 Multi-Language**: 12+ Indic languages, with the source script auto-detected
- **🔄 Bidirectional**: Supports both Indic→English and English→Indic transliteration
- **🛡️ Production Ready**: Safe file handling, atomic writes, backup support
- **📊 Structured Output**: Rich JSON format with metadata and error handling
- **⚡ Batch Processing**: Efficient processing of large files with progress tracking

## 🎯 Supported Languages

Hindi • Tamil • Telugu • Bengali • Gujarati • Kannada • Malayalam • Punjabi • Marathi • Odia • Urdu • Sanskrit ↔ English

## Install

We strongly recommend installing `indicate` inside a Python virtual environment (see [venv documentation](https://docs.python.org/3/library/venv.html#creating-virtual-environments))

**Requirements:** Python 3.13+

```bash
pip install indicate
```

## 🔧 Quick Setup

### Local Urdu lookup

The J&K electoral-roll recovery in `instate` supplies a shared Urdu token corpus.
Build the local lookup from that corpus:

```sh
export INDICATE_DATA_DIR="$PWD/indicate/data"
uv run --group train python training/build_lookup.py --lang urdu \
    --corpus ../instate/data/jk_recovery/muse_review/urdu_all_tokens/urdu.csv.gz
indicate transliterate "خان" --from urdu --engine lookup
```

The lookup returns covered spellings without an API call. Uncovered tokens remain
unresolved with `--engine lookup`. The final corpus contains 45,427 reviewed token
types and 27,221 eligible Urdu/Latin pairs. All 4,399 native types selected by the
calibrated upnaam handoff are mapped, covering 970,947 selected occurrences. The
installed table has exactly 27,221 keys and no contested or tied key.

Some pairs have human lexical support and others are model-only silver annotations.
This is not a verified pronunciation dictionary. Counts, alternatives, exclusions
and hashes live beside `urdu.csv.gz` in `provenance.json`, `validation.json` and
`transliterations.parquet`. The shared corpus contains decoded tokens only; it has
no voter IDs, source filenames, locations or relationships. Source and selection
notes are in `data/README.md`.

### For LLM-based transliteration (recommended):
```bash
pip install indicate

# Set your API key (choose one):
export OPENAI_API_KEY=your-key
export ANTHROPIC_API_KEY=your-key  
export GOOGLE_API_KEY=your-key
```

### For the local model (no API key):
```bash
pip install indicate
# No API key needed. The PyTorch weights are downloaded once from Hugging Face
# (gojiberries/indicate) on first transliterate and cached locally; tokenizers ship
# in the wheel. After the first run it works fully offline.
```

### For the lookup backend:

Kannada has a local lookup built from the existing electoral word-pair corpus in
`data/kannada.csv.gz`. Build it with
`uv run --group train python training/build_lookup.py --lang kannada`, then use
`indicate transliterate "ಶಿವಪ್ಪ" --from kannada --engine lookup`. Uncovered words
remain lookup misses; this does not invoke a paid provider. Corpus provenance is
in `data/kannada.provenance.json`.

Malayalam has a local lookup built from `data/malayalam.csv.gz` (Kerala parallel
rolls and Lakshadweep spelling harvests). Build it with:

```sh
uv run --group train python training/build_lookup.py --lang malayalam
indicate transliterate "സുഹൈൽ" --from malayalam --engine lookup
```

The corpus and harvest provenance are described in `data/malayalam.md`. The
table is built locally; lookup misses do not invoke a paid provider.


The Bengali word table downloads from the pinned model-assets repository on
first use and is then cached. It is compiled from a shared, LLM-labeled
electoral-name corpus into one deterministic native-to-Latin lookup; the
multi-million-row source CSV is not duplicated in this repository or package.

Hindi and Punjabi tables are different: they derive from
`data/hindi.csv.gz` (which blends CC-BY-NC IIT Bombay pairs) and
`data/punjabi.csv.gz` (from a restricted electoral-roll deposit), neither of
which is ours to redistribute under MIT. Build those from a checkout:

```bash
export INDICATE_DATA_DIR=~/.local/share/indicate     # where your tables live
uv run --group train python training/build_lookup.py --lang hindi
uv run --group train python training/build_lookup.py --lang punjabi
```

`INDICATE_DATA_DIR` is where the builder writes and where an installed package
looks first. Without it the table lands inside the checkout, which a
`pip install`ed copy in `site-packages` will never read. Keep it exported and
`indicate languages` flips that row from `unavailable` to `ready`:

```
Direction                 Backend   Status
bengali -> english        lookup    downloads on first use
                          llm       needs an API key
punjabi -> english        lookup    ready
                          model     ready
```

Without a Hindi or Punjabi table nothing breaks: `lookup` declines every word
and `model` answers them. Bengali is lookup-only locally, so an unavailable
table is reported as an error instead of silently returning blank text.

## 🎯 Usage

One command, one function. The language and the backend are arguments, not
separate entry points.

```bash
# Source language auto-detected from the script
indicate transliterate "राजशेखर चिंतालपति"
# rajshekhar chintalpati

indicate transliterate "ਰਵਿ ਸ਼ਰਮਾ"
# ravi sharma

indicate transliterate "বৰুৱা"
# barua

# Devanagari carries several languages and detection picks Hindi, so say it
# explicitly when it is not. Marathi has no local model — hence --engine llm
indicate transliterate "नमस्ते" --from marathi --engine llm

# Files, with the usual safety options
indicate transliterate --input names.txt --output roman.txt --format json --backup
indicate transliterate --input names.txt --output roman.txt --dry-run

# What can this install actually do?
indicate languages

# Model architecture, training sources, where the weights come from
indicate info
```

`python -m indicate` does the same as the `indicate` script, for when the
console script is not on `PATH`.

```python
import indicate

indicate.transliterate("राजशेखर चिंतालपति")  # "rajshekhar chintalpati"
indicate.transliterate("ਰਵਿ", source="punjabi")  # "ravi"
indicate.transliterate("नमस्ते", n=3)  # 3 ranked candidates
indicate.transliterate_batch(["हिंदी", "मुंबई"])  # ["hindi", "mumbai"]

indicate.supported()  # {(source, target): (backends...)}
```

### Choosing the engine

A word is answered by the first backend that will answer it. The chain is an
argument, so you decide how much machinery each word is worth:

| chain | what it does |
|---|---|
| `lookup, model` | **default** — read the table, decode the rest locally |
| `model` | decode everything; what a benchmark must use |
| `lookup` | table only, `""` on a miss — "is my corpus already covered?" |
| `lookup, llm` | the table intercepts the paid path |
| `lookup, model, llm` | escalate to a provider only what both decline |
| `llm` | ask a provider for everything |

```bash
indicate transliterate "मुंबई" --engine model
indicate transliterate "मुंबई" --engine lookup,llm --provider openai
```

```python
indicate.transliterate("मुंबई", engine=["lookup", "llm"])
indicate.transliterate("मुंबई", engine="model")
```

A backend that cannot serve a direction is skipped; if none remain you get an
error naming what would work, rather than a silent fallback onto something that
costs money:

```
$ indicate transliterate "வணக்கம்"
Error: no backend in ['lookup', 'model'] supports tamil->english;
try engine=['llm'] or see indicate.supported()
```

That is `UnsupportedPairError`. A different failure gets its own type, because
the two mean opposite things:

- a backend that **declined** — it loaded its table and had no entry for that
  word — is ordinary and silent. `engine=["lookup"]` over an uncovered corpus
  declines everything and returns `""`, which is the whole point of asking.
- a backend that was **unavailable** — no table built, no weights, no network —
  answers nothing because it could not run. When *every* backend in the chain is
  in that state you get `BackendsUnavailableError` naming each one and what to
  do about it, rather than an empty string that looks like an answer.

```python
try:
    indicate.transliterate("राजशेखर")
except indicate.BackendsUnavailableError as exc:
    print(exc)  # nothing could answer 1 word(s): lookup has no table (build ...
```

### Why the lookup backend is first by default

Known words are answered from the word table and never reach the decoder. On
Punjab electoral-roll text that covers 99.1% of tokens, so the model handles the
tail: **42x** the end-to-end throughput (10,937 tok/s against 258), and an input
that hits entirely never even imports torch, which is worth **4.4x** on cold
start (0.10s to first answer against 0.44s). `training/bench_lookup.py`
reproduces both.

It is also more accurate than either component alone, because the builder
declines to answer where the training corpus has no majority and lets those
words fall through: on the Dakshina test set, 78.8% exact against the model's
76.2% for Hindi, 77.6% against 77.0% for Punjabi.

Two caveats worth knowing before you rely on those numbers. They are measured on
**electoral-roll names**; on general Wikipedia prose the same table covers 56.9%
of tokens, not 99.1%, and the cold-start win largely disappears because a
sentence almost always contains a miss. And the shipped table contains 908 of
the 2,500 Dakshina Hindi test words, so the Hindi accuracy figure is optimistic
by an unknown amount. `training/build_lookup.py --eval-clean` builds a table
with every eval word withheld.

Use `--engine model` (or `engine=["model"]`) to measure the model by itself —
benchmarks must, or they score memorization. `training/seam_check.py` checks
that mixing table and model output in one string stays stylistically consistent.

### The LLM backend directly

For whole-sentence transliteration with context, use the client rather than the
engine chain — the chain resolves word by word:

```python
from indicate import IndicLLMTransliterator

transliterator = IndicLLMTransliterator("hindi", "english")
transliterator.transliterate("राजशेखर चिंतालपति")
transliterator.transliterate_batch(["राजेश", "गौरव", "प्रिया"])
```

For millions of tokens, `indicate.batch` uses
[batchlane](https://github.com/gojiplus/batchlane) to submit asynchronous provider
batches with checkpointing, and answers what it can locally first:

```python
from indicate.batch import transliterate_tokens_batched

pairs = transliterate_tokens_batched(
    tokens,
    "punjabi",
    "english",
    checkpoint_path="run.jsonl",
    engine=("lookup", "llm"),  # default; ("lookup","model","llm") goes further
)
```

Submission and collection can run in separate processes:

```python
from indicate.batch import (
    submit_transliteration_batches,
    collect_transliteration_batches,
)

submit_transliteration_batches(
    tokens,
    "punjabi",
    "english",
    checkpoint_path="run.jsonl",
    provider="gemini",
    model="gemini-2.5-flash",
    use_few_shot=False,
)
done, pairs = collect_transliteration_batches("run.jsonl")
```

Keep the checkpoint and its `.batchstate.json` and `.batchlane.jsonl` sidecars
together while a job is running. The state saves exact prompts; batchlane saves
provider handles. Resuming completes interrupted submissions before collecting
results. The resolved token file remains after the driver finishes, so answers
survive provider retention limits. Use one writer per checkpoint and a separate
checkpoint for each language pair and model. Provider support follows batchlane;
Azure, Vertex AI, and Bedrock batch adapters are not included.

## 📊 JSON Output Format

`--format json` works with every backend, not just the LLM. One line of input in,
one entry out, with the chain that answered it recorded per row:

```json
{
  "metadata": {
    "source_language": "hindi",
    "target_language": "english",
    "timestamp": "2026-08-14T07:40:08.697757+00:00",
    "total_lines": 1,
    "successful_lines": 1,
    "failed_lines": 0,
    "format_version": "1.0",
    "encoding": "utf-8",
    "description": "Indic language transliteration results from indicate package"
  },
  "results": [
    {
      "line_number": 1,
      "input_text": "राजेश कुमार",
      "output_text": "rajesh kumar",
      "source_lang": "hindi",
      "target_lang": "english",
      "confidence": "lookup,model",
      "error": null,
      "processing_time": 0.07029390335083008,
      "timestamp": "2026-08-14T07:40:08.697423+00:00"
    }
  ]
}
```

`confidence` holds the engine chain, not a probability — the local model's beam
scores are not calibrated, so publishing one would invite a comparison it cannot
support.

## 🛡️ Safety Features

- **🔒 Input/Output Validation**: Prevents accidental file overwrites
- **⚛️ Atomic Writing**: Safe file operations using temporary files
- **💾 Automatic Backups**: Optional timestamped backups of existing files
- **👁️ Dry Run Mode**: Preview operations before execution

Resumable runs live in `indicate.batch`, which checkpoints every resolved token
to disk and picks up where it left off.

## 🎛️ Advanced Usage

```bash
# Pick an LLM provider and model
indicate transliterate "text" --engine llm --provider anthropic --model claude-3-opus

# Read JSON produced by an earlier run
indicate transliterate --input results.json --from english --to hindi --engine llm

# Table only: how much of this file does the table already cover?
indicate transliterate --input names.txt --engine lookup
```

## 🔄 Backend Comparison

| | `lookup` | `model` | `llm` |
|---|---|---|---|
| **Directions** | Bengali, Hindi, Punjabi → English | Hindi, Punjabi → English | 12+ languages, any Indic pair |
| **Setup** | Bengali downloads; build Hindi/Punjabi | none | API key |
| **Speed** | 10,937 tok/s end to end | 258 tok/s | network-bound |
| **Cost** | free | free | per API call |
| **Offline** | ✅ | ✅ | ❌ |
| **Coverage** | only what is in the table | every word | every word |
| **Answers with** | the corpus label | a decode | the provider |

Both speeds are end-to-end on roll names, measured back to back on one machine,
so the ratio is the meaningful part. The table itself serves 16.9M reads/s once
loaded; that number describes the dictionary, not the pipeline, and quoting it
as throughput would overstate the win by three orders of magnitude.

`indicate languages` prints which of these are available for a direction on your
machine.

## 🧪 Testing Locally

1. **Clone and install**:
   ```bash
   git clone https://github.com/in-rolls/indicate.git
   cd indicate
   uv sync  # or pip install -e .
   ```

2. **Run tests**:
   ```bash
   uv run pytest                       # everything
   uv run pytest tests/test_engine.py  # one file
   ```

   Model weights and lookup tables are gitignored, so a fresh clone skips the
   tests that need them and prints what is missing with the command that builds
   it. To make those skips into failures instead — which is what CI does, after
   building the tables from the committed corpora:

   ```bash
   uv run pytest --require-artifacts
   ```

3. **Test the backends**:
   ```bash
   # Local, no API key
   indicate transliterate "हिंदी" --engine lookup,model

   # LLM (set an API key first)
   export OPENAI_API_KEY=your-key
   indicate transliterate "हिंदी" --engine llm
   ```

## Data

The datasets used to train the model:

- [Indian Election affidavits](https://affidavit.eci.gov.in/CandidateCustomFilter)
- [Google Dakshina dataset](https://github.com/google-research-datasets/dakshina)
- [ESPN Cric Info](https://www.espncricinfo.com/hindi/series/pakistan-tour-of-england-2021-1239529/england-vs-pakistan-1st-odi-1239537/full-scorecard) for hindi version of the [english scorecard](https://www.espncricinfo.com/series/pakistan-tour-of-england-2021-1239529/england-vs-pakistan-1st-odi-1239537/full-scorecard)
- [IIT Bombay English-Hindi Corpus](https://www.cfilt.iitb.ac.in/iitb_parallel/)

## Evaluation

The v2 models (trained on our data + the public [Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar)
corpus) are benchmarked against **AI4Bharat IndicXlit** — the same direction
(native→Latin), the same test sets, the same metric (Top-1 exact-match,
match-any-reference). Training is leakage-filtered so no eval word appears in it.

| Model | Dakshina (gold) | Held-out-own names¹ |
|-------|-----------------|---------------------|
| Hindi → English | **74.4%** (IndicXlit 73.2%) | **52.8%** (IndicXlit 49.7%) |
| Punjabi → English | 71.9% (IndicXlit 73.2%) | **56.9%** (IndicXlit 53.5%) |

¹ Held-out slice of our own electoral/affidavit names — the cleanest comparison,
since IndicXlit never trained on it. **v2 matches or edges IndicXlit on the gold
benchmark and beats it on the deployment domain.** Primary metric is Top-1
exact-match; CER (character error rate) is the soft companion. Reproduce with
`training/eval.py` and `training/compare.py`.

Below is the edit-distance distribution on the test set (0 = exact match):

![Edit distance metrics of model on Google Dakshina test dataset](https://github.com/in-rolls/indicate/raw/master/images/h2e_ed.png)

## Authors

Rajashekar Chintalapati and Gaurav Sood

## Contributor Code of Conduct

The project welcomes contributions from everyone! In fact, it depends on it. To maintain this welcoming atmosphere, and to collaborate in a fun and productive way, we expect contributors to the project to abide by the [Contributor Code of Conduct](http://contributor-covenant.org/version/1/0/0/).

## License

The package is released under the [MIT License](https://opensource.org/licenses/MIT).
