Metadata-Version: 2.4
Name: sanskrit-tokenizer
Version: 0.2.1
Summary: Sanskrit tokenizer with sandhi splitting for Information Retrieval.
Author: Hemanth HM
License-Expression: MIT
Project-URL: Homepage, https://github.com/hemanth/sanskrit-tokenizer
Project-URL: Repository, https://github.com/hemanth/sanskrit-tokenizer
Keywords: sanskrit,tokenizer,sandhi,nlp,iast,devanagari
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# sanskrit-tokenizer

Tokenize Sanskrit text with sandhi splitting for Information Retrieval.

```bash
pip install sanskrit-tokenizer
```

## Quick start

```python
from sanskrit_tokenizer import tokenize

tokenize("dharmakṣetre kurukṣetre")
# ['dharmakṣetre', 'kurukṣetre']

tokenize("धर्म योग")
# ['धर्म', 'योग']

tokenize("devālaya namaḥ")
# ['deva', 'ālaya', 'namaḥ']
```

`tokenize()` accepts both Devanagari and IAST input — output script matches input. Splits on whitespace and punctuation, applies reverse sandhi rules, and preserves the original script.

## Sandhi splitting

```python
from sanskrit_tokenizer.sandhi import split_sandhi

split_sandhi("devālaya")     # deva(832) + ālaya(76) — both in dict ✓
# ['deva', 'ālaya']

split_sandhi("narottama")    # nara(187) + uttama(160) — guṇa: a + u → o
# ['nara', 'uttama']

split_sandhi("dharmakṣetre")  # kṣa(0) + itre(0) — not real words ✗
# ['dharmakṣetre']
```

Dictionary-aware scoring: splits are validated against an embedded 8,750-word Sanskrit frequency dict (from Wiktionary, CC BY-SA). Candidates producing real words score dramatically higher than spurious fragments. Rule engine covers vowel sandhi (savarṇa-dīrgha, guṇa, vṛddhi, yān, ayādi), consonant sandhi, and visarga sandhi.

## Transliteration

```python
from sanskrit_tokenizer.transliterate import (
    devanagari_to_iast,
    iast_to_devanagari,
    is_devanagari,
)

devanagari_to_iast("संस्कृत")
# 'saṃskṛta'

iast_to_devanagari("dharma")
# 'धर्म'

is_devanagari("संस्कृत")
# True
```

## Word-level tokenization

```python
from sanskrit_tokenizer import tokenize_words

tokenize_words("dharma yoga namaḥ")
# ['dharma', 'yoga', 'namaḥ']

tokenize_words("धर्म योग")
# ['धर्म', 'योग']
```

`tokenize_words()` splits on whitespace and punctuation only — no sandhi splitting.

## CLI

```bash
sanskrit-tokenize "dharma yoga"
# dharma
# yoga

sanskrit-tokenize "धर्म योग"
# धर्म
# योग

sanskrit-tokenize --no-sandhi "devālaya"
# devālaya

sanskrit-tokenize -s " " "dharma yoga"
# dharma yoga

sanskrit-tokenize --version
# sanskrit-tokenize 0.2.0
```

- `--no-sandhi` — word-level only, skip sandhi splitting
- `--separator SEP` — output separator (default: newline)
- `--version` — show version

## License

MIT © [Hemanth.HM](https://h3manth.com)
