Metadata-Version: 2.5
Name: qasd
Version: 0.0.1
Summary: Qasd (قصد): understanding what Arabic shoppers mean, in any dialect. Normalization, dialect-aware retrieval and evaluation.
Project-URL: Repository, https://github.com/esammahdi/QasdSearch
Author: Esameldin Bashir
License-Expression: Apache-2.0
Keywords: arabic,dialects,embeddings,nlp,query-understanding,retrieval,search
Classifier: Natural Language :: Arabic
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Provides-Extra: dense
Requires-Dist: sentence-transformers>=6.1; extra == 'dense'
Requires-Dist: torch>=2.4; extra == 'dense'
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Provides-Extra: gen
Requires-Dist: openai>=1.40; extra == 'gen'
Provides-Extra: train
Requires-Dist: accelerate>=1.0; extra == 'train'
Requires-Dist: datasets>=3.0; extra == 'train'
Requires-Dist: sentence-transformers>=6.1; extra == 'train'
Requires-Dist: torch>=2.4; extra == 'train'
Description-Content-Type: text/markdown

# Qasd (قصد)

**Understanding what Arabic shoppers mean, in any dialect.**

Type "iPhone" into an Arabic store and people write it six ways: آيفون, أيفون, ايفون, ايفونه, iphone, ayfon. Add a model number and you get ١٥ next to 15. Ask for shoes and an Egyptian types جزمة, someone from the Gulf types جوتي, someone from the Levant types صرماية, and the catalog says حذاء.

A search engine has to understand all of them. Qasd is a small, honest benchmark and toolkit for that problem: Arabic **query understanding** (normalization, dialects, Arabizi, code-switching) and **retrieval** (BM25, multilingual embeddings, hybrid search), measured per dialect and per error type.

## What's here
| Piece | What it does |
|---|---|
| `qasd.normalize` | Configurable Arabic normalizer: alef variants, ة/ه, ى/ي, hamza carriers, diacritics, tatweel, Arabic-Indic digits. Every rule is a switch, so its value can be measured. |
| `qasd.arabizi` | Baseline Arabizi detector and transliterator (sha7en → شاحن). |
| `qasd.retrieval` | BM25 (pure Python), dense retrieval with EmbeddingGemma 2 via sentence-transformers, and reciprocal-rank fusion. |
| `qasd.eval` | Recall@k, MRR, nDCG@k, reported overall, **by dialect** and **by phenomenon**. |
| `scripts/generate_queries.py` | LLM-generated dialect queries from product titles, flagged for human review. |
| `notebooks/finetune_colab.ipynb` | Fine-tunes EmbeddingGemma 2 on Arabic query → product pairs on a Colab GPU. |

## Experiments
1. **Baselines:** BM25 with and without normalization, then EmbeddingGemma 2 out of the box.
2. **Fine-tuning:** EmbeddingGemma 2 trained on Arabic dialect query → product pairs, before vs after.
3. **Matryoshka sizes:** how much accuracy 768 → 256 → 128 dimensions costs, against the speed and memory saved.
4. **Spoken queries:** typed vs speech-to-text vs EmbeddingGemma 2's direct audio embeddings. Does the voice path return the same products as the typed one?

Results are reported per dialect and per error type, because an average hides exactly the cases this project is about.

## Results
Nothing yet beyond the pipeline check on `data/sample` (18 synthetic queries). The real eval set is 100–200 hand-checked pairs; see [data/README.md](data/README.md).

## Quickstart
```bash
uv run --extra dev pytest -q                        # tests, no heavy installs
uv run scripts/run_eval.py --retriever bm25 --show-misses
uv run scripts/run_eval.py --retriever bm25 --no-normalize
uv run --extra dense scripts/run_eval.py --retriever dense   # needs torch + the model
```

## Notes
- EmbeddingGemma 2 must run in **bfloat16 or float32, never float16**. `qasd.retrieval.dense.pick_dtype()` handles this.
- All data is synthetic or public; no real customer queries. Every query records whether it was hand-written, LLM-generated or human-checked.

See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), [docs/EXPERIMENTS.md](docs/EXPERIMENTS.md) and [docs/ROADMAP.md](docs/ROADMAP.md).

License: Apache-2.0.
