Metadata-Version: 2.4
Name: darija-router
Version: 0.15.0
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Natural Language :: Arabic
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
License-File: LICENSE
Summary: Moroccan Darija NER and semiotic class tagging — rule-based NER engine that outperforms GPT-4o-mini on Darija text. Built in Rust for O(1) lookup, 750x faster than LLMs.
Keywords: nlp,ner,arabic,darija,moroccan,semiotic,code-switching,rule-based
Author-email: Ouail Laamiri <laamiri.ouail@etu.uae.ac.ma>, Ismail Berrada <ismail.berrada@um6p.ma>
License: MIT
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Repository, https://github.com/LaamiriOuail/darija-router

# darija-router

[![PyPI version](https://img.shields.io/pypi/v/darija-router.svg)](https://pypi.org/project/darija-router/)
[![CI](https://github.com/LaamiriOuail/darija-router/actions/workflows/ci.yml/badge.svg)](https://github.com/LaamiriOuail/darija-router/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python](https://img.shields.io/pypi/pyversions/darija-router.svg)](https://pypi.org/project/darija-router/)

Moroccan Darija NER and semiotic class tagging — rule-based, zero-training, O(1) lookup. Built in Rust, wrapped for Python.

**63.0% F1 on Darija NER — outperforms GPT-4o-mini (57.0%) at 750x the speed.**

## Benchmark

Evaluated on 500 Moroccan Darija sentences (998 annotations, 12 entity types). All LLM baselines zero-shot via OpenRouter API.

| System | Precision | Recall | F1 | Speed |
|--------|-----------|--------|----|-------|
| **darija-router** | **57.8%** | **69.3%** | **63.0%** | **1.2s** |
| GPT-4o-mini | 62.9% | 52.2% | 57.0% | ~15min |
| GPT-4o | 64.6% | 43.2% | 51.8% | ~20min |
| Qwen3.6-35B | 61.4% | 23.1% | 33.6% | ~45min |
| GPT-5-mini | 65.9% | 8.9% | 15.7% | ~10min |

darija-router runs on a single CPU core. No GPU. No API calls. No training data.

### Ablation Study

| Configuration | P | R | F1 |
|---|---|---|---|
| Full system | 57.8% | 69.3% | **63.0%** |
| − Lexicon lookup | 57.8% | 69.3% | 63.0% |
| − Morphological analysis | 57.8% | 69.3% | 63.0% |
| − NER triggers (PER/LOC/ORG) | 57.6% | 70.0% | 63.2% |
| − Temporal patterns | 62.0% | 59.7% | 60.9% |
| − All semantic tagging | 0.0% | 0.0% | 0.0% |

See [eval/README.md](eval/README.md) for full methodology and per-label results.

## Install

```bash
pip install darija-router
```

Requires Python ≥ 3.8. Wheels available for Linux, macOS, and Windows (x86_64 and ARM64).

## Quick Start

```python
import darija_router

# NER — extract named entities
result = darija_router.run_ner("شريت 3 كيلو ديال التفاح ب 20 درهم ليوم")
print(result["entities"])
# [{"text": "3 كيلو", "entity_type": "MEASURE", ...},
#  {"text": "20 درهم", "entity_type": "MONEY", ...},
#  {"text": "ليوم", "entity_type": "DATE", ...}]

# Filter to specific entity types
result = darija_router.run_ner(
    "شريت 3 كيلو ب 20 درهم",
    labels=["MONEY", "MEASURE"]
)
print(result["entities"])
# [{"text": "3 كيلو", "entity_type": "MEASURE", ...},
#  {"text": "20 درهم", "entity_type": "MONEY", ...}]

# Code-switching detection
cs = darija_router.run_codeswitch("kifash nchri car b 5000 euro?")
print(cs["stats"]["darija"])  # 0.4
print(cs["stats"]["fr"])      # 0.2

# Full analysis
result = darija_router.analyze("خويا باغي نلقاك غدوة الساعة 3 صباح")
```

## API Reference

### `run_ner(phrase, labels=None, config=None)`

Named Entity Recognition. Returns span-based entities and per-token NER tags.

**Parameters:**
- `phrase` (str): Input text (Arabic script or Arabizi)
- `labels` (list[str], optional): Filter output to only these entity types. Default: all types. Useful for evaluation when the gold schema excludes PERSON, LOC, etc.
- `config` (dict, optional): Pipeline configuration for ablation studies. Keys: `disable_lexicon`, `disable_morphology`, `disable_semantic`, `disable_temporal`, `disable_ner_triggers`. All default to `False`.

**Returns:**
```json
{
  "entities": [
    {"text": "500 درهم", "entity_type": "MONEY", "start": 0, "end": 8,
     "token_indices": [0, 1], "confidence": 0.95}
  ],
  "tokens": [
    {"token": "500", "ner_tag": "MONEY"},
    {"token": "درهم", "ner_tag": "MONEY"}
  ]
}
```

### `analyze(phrase, mode=None, fields=None)`

Tokenizes a phrase and classifies each token.

**Parameters:**
- `phrase` (str): Input text in Arabic script
- `mode` (str, optional): Output preset — `"full"` (default), `"compact"`, `"ner"`, `"codeswitch"`
- `fields` (list[str], optional): Custom field selection — e.g. `["token", "lang", "confidence"]`

**Returns:** `list[dict]` with per-token results.

### `run_codeswitch(phrase)`

Code-switching detection with language spans and statistics.

**Returns:**
```json
{
  "spans": [{"text": "خويا واش", "lang": "darija", "start": 0, "end": 9}],
  "tokens": [{"token": "خويا", "lang": "darija", "latin": null, "confidence": 1.0}],
  "stats": {"darija": 50.0, "fr": 25.0, "en": 0.0, "num": 12.5, "punct": 0.0, "other": 0.0, "unknown": 12.5}
}
```

### `lookup(word)`

Single word lookup. Same format as one element from `analyze()`.

### `normalize_arabic(text)` / `arabizi_to_arabic(text)` / `normalize(text)`

Normalization functions. `normalize()` auto-detects script and applies the appropriate function.

### `tokenize(text)`

Tokenize text into tokens with character offsets.

### `detect_script(text)`

Detect script: `"arabic"`, `"latin"`, `"mixed"`, or `"unknown"`.

### `__version__`

Package version string (e.g. `"0.15.0"`).

## Entity Types

| Type | Detection | Example |
|------|-----------|---------|
| `CARDINAL` | Digits or Arabic number words | `50` `عشرين` `مية` |
| `ORDINAL` | Arabic ordinal words (standard + Darija) | `تاني` `تالت` `اللول` |
| `DECIMAL` | Decimal numbers | `3.5` `١٢.٥` |
| `MONEY` | Cardinal + currency word | `500 درهم` |
| `MEASURE` | Cardinal + unit word | `2 كيلو` `24 ساعة` |
| `TIME` | Clock times, periods, durations | `الساعة 3` `صباح` `8 د الصباح` |
| `DATE` | Date words, years, day names, durations | `غدوة` `15 مارس` `2026` |
| `PHONE` | Phone number patterns (Moroccan) | `+212661234567` `0661234567` |
| `EMAIL` | Email addresses | `user@gmail.com` |
| `URL` | URLs | `http://google.com` |
| `PERCENT` | Percentage suffixes | `20%` `بالمئة` |
| `ADDRESS` | Address keywords + following word | `زنقة الحسن` |
| `PERSON` | Honorifics + name expansion | `سيدي محمد` `لالة فاطمة` |
| `LOC` | Location triggers + expansion | `شارع محمد الخامس` `مدينة الدار البيضاء` |
| `ORG` | Organization triggers + expansion | `شركة مسير` `بنك الشعب` |
| `FACILITY` | Facility triggers + expansion | `سبيسان` `مسجد الكبير` |
| `EVENT` | Event triggers + expansion | `عيد الفطر` `عرس` |
| `MISC` | Nationalities, languages, religions | `المغربي` `الفرنساوي` `الإسلام` |
| `FRACTION` | Arabic fraction words | `نص` `ربع` `تلت` |

## Semantic Tags

Semantic tags are internal per-token labels that feed into the NER engine. They correspond to entity types above but may include intermediate labels like `PUNCT`.

| Tag | Detection | Example |
|-----|-----------|---------|
| `PUNCT` | Punctuation characters | `؟` `،` `.` |
| `CARDINAL` | Digits or Arabic number words | `50` `عشرين` `مية` |
| `ORDINAL` | Arabic ordinal words | `تاني` `تالت` `اللول` |
| `MONEY` | Cardinal + currency word | `50 درهم` → both tokens |
| `MEASURE` | Cardinal + unit word | `2 كيلو` → both tokens |
| `TIME` | Time patterns | `الساعة 3` `صباح` |
| `DATE` | Date patterns | `غدوة` `15 مارس` |
| `FRACTION` | Fraction words | `نص` `ربع` |
| `PERCENT` | Percentage suffix | `20%` |
| `PHONE` | Phone patterns | `+212...` |
| `EMAIL` | Email patterns | `user@...` |
| `URL` | URL patterns | `http://...` |
| `ADDRESS` | Address keywords | `زنقة` `شارع` |

## Output Modes

| Mode | Description |
|------|-------------|
| `"full"` (default) | All fields: token, index, lang, entries, stem, semantic, confidence |
| `"compact"` | token + lang only (no entries/stem) |
| `"ner"` | NER preset: entities + tokens with ner_tag |
| `"codeswitch"` | Code-switching preset: spans + tokens + stats |

## Confidence Scoring

| Match Type | Score |
|------------|-------|
| Exact match | 1.0 |
| Normalized (ة→ه) | 0.95 |
| Single prefix/suffix strip | 0.85 |
| Double strip (prefix + suffix) | 0.70 |
| Semantic tag only (no lexicon match) | 0.60 |
| Unknown | 0.0 |

## Pipeline Architecture

darija-router processes text through a six-phase pipeline:

1. **Script Detection** — classify each token as Arabic, Latin, Mixed, or Numeric
2. **Normalization** — Arabizi (Latin script) → Arabic script using dictionary + heuristics (427 exceptions)
3. **Morphological Analysis** — strip Arabic prefixes (7) and suffixes (5) withت→ه normalization
4. **Lexicon Lookup** — O(1) HashMap lookup against 2,187 embedded entries, with morphological fallback
5. **Semantic Tagging** — context-aware rule engine detecting 14+ categories with multi-token grouping
6. **NER Extraction** — trigger-based detection for PERSON, LOC, ORG, FACILITY, EVENT, MISC with expansion rules

All lexicons compile into the binary via `include_str!()`. Zero runtime file I/O. Zero startup cost. 3.2 MB binary.

## Temporal Expression Recognition

darija-router recognizes 12 distinct Darija temporal patterns:

| Pattern | Example | Structure |
|---------|---------|-----------|
| Clock time | `الساعة 8` | article + number |
| Prepositional | `8 د الصباح` | number + preposition + period |
| Contracted | `12 دالليل` | number + attached preposition |
| Duration hours | `24 ساعة` | number + hour unit |
| Duration minutes | `15 دقيقة` | number + minute unit |
| Fraction hour | `نص ساعة` | fraction + unit |
| Clock with fraction | `6 ونص` | number + connector + fraction |
| Midnight | `منتصف الليل` | fixed phrase |
| Noon | `منتصف النهار` | fixed phrase |
| Period only | `8 مساء` | number + period word |
| Definite hour | `السبعة` | article + number word |
| HH:MM | `14:30` | direct time notation |

This specialized handling improved TIME F1 from 0% to 38.1%.

## Build from Source

```bash
git clone https://github.com/LaamiriOuail/darija-router.git
cd darija-router
pip install maturin
maturin develop --release
```

## Testing

```bash
pip install pytest
pytest tests/ -v
```

695+ tests covering tokenization, normalization, morphology, semantic tagging, NER, code-switching, and span positions.

## Citation

If you use darija-router in your research, please cite:

```bibtex
@software{darija_router_2026,
  author = {Laamiri, Ouail and Berrada, Ismail},
  title = {darija-router: Rule-Based NER for Moroccan Darija},
  year = {2026},
  url = {https://github.com/LaamiriOuail/darija-router},
  note = {63.0\% F1 on Darija NER, outperforming GPT-4o-mini at 750x speed}
}
```

## License

MIT
