Metadata-Version: 2.5
Name: hindko-embeddings
Version: 1.0.0
Summary: Hindko text embeddings with a small Python SDK: semantic search, similarity, clustering and classification for Hindko.
Project-URL: Model card, https://huggingface.co/junaid008/hindko-embeddings
Project-URL: Tokenizer, https://pypi.org/project/hindko-tokenizer/
Author: Junaid Ahmed
License-Expression: CC-BY-NC-4.0
License-File: LICENSE
Keywords: embeddings,hindko,low-resource,nlp,pakistan,rag,semantic-search,sentence-transformers,urdu-script
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: Urdu
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: hindko-tokenizer>=1.0.0
Requires-Dist: sentence-transformers>=6.0
Requires-Dist: transformers>=5.0
Provides-Extra: test
Requires-Dist: pytest>=7; extra == 'test'
Description-Content-Type: text/markdown

# hindko-embeddings

Hindko text embeddings as a Python package. One install gives you semantic search, similarity, clustering and classification for Hindko text.

Hindko (Northern `hno`, Southern `hnd`) is the Indo-Aryan language of Peshawar, Hazara and Kohat in Khyber Pakhtunkhwa, Pakistan, written in Urdu (Perso-Arabic) script. The model turns a Hindko headline, sentence, verse or passage into a 1,024-dimensional vector, so that texts about the same thing end up close together.

```bash
pip install hindko-embeddings
```

![Scorecard](https://huggingface.co/junaid008/hindko-embeddings/resolve/main/benchmarks/00_scorecard.png)

## Quick start

```python
from hindko_embeddings import HindkoEmbedder

embedder = HindkoEmbedder.from_pretrained()   # first run downloads the model (1.1 GB), then it is cached

passages = [
    "ایہہ بہت کہٹ لوک جانڑدین کہ ڈرامے دی پیدائش کسراں تے کتھے ہوئی آئی۔",
    "اج اساں اگر آپڑیں بزرگاں دی انہاں گلاں تے غور نہ کیتا تے پچھتاوے دے سوا کج ہتھ نہ آسی۔",
    "اسی طراں خیبر پختونخوا اچ بولی جانڑیں ولی دیگر زباناں دے ادب و ثقافت دی بی نمائندگی کیتی گئی اَئی۔",
]

for hit in embedder.search("ڈرامے دی پیدائش", passages, top_k=3):
    print(round(hit["score"], 3), hit["text"])
```

The SDK applies the two conventions the model was trained with, so you don't have to:

- it puts text in canonical form with `normalize()` from [`hindko-tokenizer`](https://pypi.org/project/hindko-tokenizer/), which is installed with this package;
- it adds the `query: ` or `passage: ` prefix the model expects.

## What you can do

```python
q = embedder.encode_queries(["ڈرامے دی پیدائش"])   # search queries, headlines, titles
p = embedder.encode_passages(passages)             # documents to be found
scores = embedder.similarity(q, p)                 # cosine similarity matrix

emb = embedder.encode(texts)                       # similarity, clustering, classification ("query" side)
vector = embedder.encode("ڈرامے دی پیدائش")         # one string gives one vector

hits = embedder.search(query, passages, top_k=5)                         # ranked [{index, score, text}]
hits = embedder.search(query, passages, passage_embeddings=p)            # reuse embeddings across queries

embedder.prepare(texts, kind="passage")   # see exactly what the model receives
embedder.model                            # the underlying SentenceTransformer
```

`from_pretrained()` takes `device="cuda"`, a pinned `revision=` and a `cache_folder=`. On a CPU the float16 weights are cast to float32 automatically.

## Results

On the held-out test split of HindkoEmbedBench v1.0 (five tasks: headline retrieval, book-title retrieval, passage continuation, verse/prose classification and book clustering):

| Model | Test score |
|---|---:|
| **hindko-embeddings** | **0.7391** |
| Character n-gram TF-IDF | 0.6864 |
| bge-m3 (best public embedding model tested) | 0.6332 |
| multilingual-e5-large (its base model, not fine-tuned) | 0.5977 |

It has the highest overall test score of the 12 models scored on test, and it is ahead of TF-IDF on four of the five tasks (behind it on book-title retrieval). Every number, chart and the full protocol are on the [model card](https://huggingface.co/junaid008/hindko-embeddings).

## Requirements

Python 3.10 or later, sentence-transformers 6.0 or later and transformers 5.x. Older versions may not load the saved model configuration. Input longer than 512 tokens is truncated.

## Links

- Model card, benchmark and evaluation: [huggingface.co/junaid008/hindko-embeddings](https://huggingface.co/junaid008/hindko-embeddings)
- Hindko tokenizer: [`hindko-tokenizer`](https://pypi.org/project/hindko-tokenizer/)

## License

[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (non-commercial): open for research, teaching and other non-commercial use. Commercial use needs written permission from the author. The base model, intfloat/multilingual-e5-large, is MIT-licensed; its notice is kept in `LICENSE`.
