Metadata-Version: 2.5
Name: hindko-tokenizer
Version: 1.0.0
Summary: Hindko tokenizer (SentencePiece Unigram, 32,768 ids) with a small Python SDK: tokenize, encode, decode and normalize Hindko text.
Project-URL: Model card, https://huggingface.co/junaid008/hindko-tokenizer
Project-URL: Source, https://github.com/junaidaslam2006/Hindko-Tokenizer
Author: Junaid Ahmed
License-Expression: CC-BY-4.0
License-File: LICENSE
Keywords: hindko,low-resource,nlp,pakistan,perso-arabic,sentencepiece,tokenizer,urdu-script
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: Urdu
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Requires-Dist: tokenizers>=0.15
Provides-Extra: test
Requires-Dist: pytest>=7; extra == 'test'
Provides-Extra: transformers
Requires-Dist: transformers>=4.40; extra == 'transformers'
Description-Content-Type: text/markdown

# hindko-tokenizer

The Hindko tokenizer as a Python package. Install it from PyPI and start tokenizing: no repository clone and no download, because the tokenizer file ships inside the package.

Hindko (Northern `hno`, Southern `hnd`) is the Indo-Aryan language of Peshawar, Hazara and neighbouring districts of Pakistan, written in Urdu (Perso-Arabic) script. This is a 32,768-id SentencePiece Unigram tokenizer trained on Hindko newspapers, books and web text.

```bash
pip install hindko-tokenizer
```

![Scorecard](https://huggingface.co/junaid008/hindko-tokenizer/resolve/main/benchmarks/00_scorecard.png)

## Quick start

```python
from hindko_tokenizer import HindkoTokenizer

tokenizer = HindkoTokenizer.from_pretrained()

text = "ایہہ بہت کہٹ لوک جانڑدین کہ ڈرامے دی پیدائش کسراں تے کتھے ہوئی آئی۔"
print(tokenizer.tokenize(text))
print(tokenizer.encode(text))
print(tokenizer.decode(tokenizer.encode(text)))   # gives back the same text
```

## What you can do

```python
tokenizer.tokenize(text)          # token strings; "▁" marks a preceding space
tokenizer.encode(text)            # token ids (no special tokens are added)
tokenizer.decode(ids)             # ids back to text
tokenizer.count_tokens(text)      # how many tokens a text costs

tokenizer.encode_batch(texts)     # many texts at once (parallel)
tokenizer.decode_batch(list_of_ids)

tokenizer.vocab_size              # 32768
tokenizer.token_to_id("\n")       # 64
```

### Normalizing input

The tokenizer itself changes nothing in your text. User-typed or pasted text often has extra spaces, invisible characters or Arabic-keyboard letters (ي ك instead of ی ک), and these cost extra tokens. `normalize()` puts text into the form the tokenizer was trained on:

```python
from hindko_tokenizer import normalize

clean = normalize(raw_text)
ids = tokenizer.encode(raw_text, normalize=True)               # same as encode(normalize(raw_text))
tokenizer = HindkoTokenizer.from_pretrained(normalize=True)    # or switch it on for every call
```

`normalize()` is deterministic and idempotent. It is not reversible, so after it `decode()` returns the normalized text. Do not use it where whitespace must stay exactly as it was. It is the frozen release `1.0.1` used for all training and evaluation text.

### Use with Hugging Face transformers

```bash
pip install "hindko-tokenizer[transformers]"
```

```python
hf_tokenizer = tokenizer.to_transformers()
hf_tokenizer(text)["input_ids"]   # the same ids as tokenizer.encode(text)
```

### Load from a file

```python
tokenizer = HindkoTokenizer.from_pretrained("path/to/folder")   # a folder with tokenizer.json, or the file itself
tokenizer.backend                                               # the underlying tokenizers.Tokenizer
```

## Results

Measured on the held-out test split of the Hindko corpus (details and every source on the [model card](https://huggingface.co/junaid008/hindko-tokenizer)):

| | |
|---|---|
| **External comparison** | Fewest tokens of the 131 distinct lossless tokenizers compared |
| **Nearest external tokenizer** | ProximaAI/urnova-95m: this tokenizer uses 16.7 % fewer tokens |
| **Frontier tokenizers** | 36.6 % fewer tokens than Gemma 3/4, 40.4 % fewer than GPT-4o, 48.8 % fewer than Llama 4, 50.6 % fewer than Qwen 3.5, 53.7 % fewer than DeepSeek-V3 |

These gains apply to a model that is trained or adapted with this tokenizer. The tokenizer cannot be plugged into an existing model such as GPT-4o or Llama.

## Links

- Model card, benchmarks and evaluation: [huggingface.co/junaid008/hindko-tokenizer](https://huggingface.co/junaid008/hindko-tokenizer)
- Source code: [github.com/junaidaslam2006/Hindko-Tokenizer](https://github.com/junaidaslam2006/Hindko-Tokenizer)
- Hindko text embeddings: [`hindko-embeddings`](https://pypi.org/project/hindko-embeddings/)

## License

[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/): free to use, share and adapt, including commercially, with attribution. The license covers the tokenizer and this package, not the texts it was trained on.
