Metadata-Version: 2.4
Name: emakhuwa-tokenizer
Version: 0.1.0
Summary: Rule-based morphological tokenizer for Emakhuwa (Bantu P.31)
Author-email: Felermino Ali <felerminoali@gmail.com>
License-Expression: MIT
Keywords: Emakhuwa,Makhuwa,Bantu,morphology,tokenizer,NLP
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rapidfuzz>=3.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Dynamic: license-file

# emakhuwa-tokenizer

emakhuwa-tokenizer/
├── AGENTS.md, BOOTSTRAP.md, HEARTBEAT.md, IDENTITY.md   # workspace/agent docs
├── SOUL.md, TASK.md, TOOLS.md, USER.md, README.md
├── LICENSE
├── config.yaml
├── pyproject.toml, pytest.ini                           # project/build config
│
├── analysis/                                            # linguistic data
│   ├── morpheme_inventory.json
│   ├── noun_class_table.json
│   └── verb_paradigms.json
│
├── evaluation/                                          # eval results & gold sets
│   ├── error_analysis.md
│   ├── gold_errors.json / gold_standard.json
│   ├── gold_standard_results.json / results.json
│   └── pytest_output.txt
│
├── reference/
│   └── MORPHOLOGY_REFERENCE.md
│
├── rules/                                               # morphology rule definitions
│   ├── extensions.json, imbrication.json
│   ├── morphophonology.json, negation.json
│   ├── noun_classes.json, spelling_normalization.json
│   └── verb_template.json
│
├── scripts/
│   ├── evaluate.py, evaluate_gold.py
│   └── phase1_analyze.py
│
├── src/emakhuwa_tokenizer/                              # main package
│   ├── __init__.py, cli.py, tokenizer.py
│   ├── decision.py, glosser.py, imbrication.py
│   ├── lemmatizer.py, lexicon.py, lookup.py
│   ├── models.py, morphophonology.py, normalizer.py
│   ├── rules.py, segmenter.py, sentence.py
│   └── rule_data/                                       # packaged copies of rules/*.json
│       ├── __init__.py
│       └── *.json (extensions, imbrication, morphophonology, ...)
│
└── tests/
    └── test_tokenizer.py

    
Rule-based morphological tokenizer for **Emakhuwa** / **Makhuwa** (Bantu P.31), focused on morphologically informed segmentation, root identification, and word-class prediction.

The package implements a transparent linguistic pipeline for Emakhuwa morphology: spelling normalization, morphophonological reversal, template-based verb parsing, noun-class analysis, extension handling, imbrication, and lexicon-assisted segmentation.

## Linguistic background

Emakhuwa is a Bantu language of Mozambique with rich agglutinative morphology. A useful morphological tokenizer must handle phenomena such as:

- noun-class prefixes: `mu-`, `a-`, `mi-`, `ma-`, `e-`, `i-`, locatives, and agentive patterns
- verbal template slots: `NEG-SM-TAM-OP-ROOT-EXT-FV`
- subject/object marker allomorphy
- TAM markers such as `naa-`, `ho-`, `nni-`
- derivational extensions: applicative, causative, reciprocal, passive, stative, intensive
- vowel-height harmony in extensions
- imbrication in recent-past forms
- spelling and apostrophe normalization

## Installation

```bash
pip install emakhuwa-tokenizer
```

For development:

```bash
git clone <repo-url>
cd emakhuwa-tokenizer
pip install -e .[dev]
pytest
```

## Quick start

```python
from emakhuwa_tokenizer import tokenize, tokenize_sentence, lemmatize, load_tokenizer

analysis = tokenize("kinaavaha")
print(analysis.root)
print([(m.form, m.slot, m.gloss) for m in analysis.morphemes])

# Reuse a tokenizer instance for many words
tok = load_tokenizer()
for word in ["kinaavaha", "nkivahale", "kivanje"]:
    print(tok.analyze_word(word).to_dict())

print(lemmatize("kinaavaha"))  # ovaha

results = tokenize_sentence("Mwana anaavaha mwalimu ehuku")
for r in results:
    print(f"{r.surface_form}: {r.root} ({r.word_class})")
```

Sentence example:

```python
>>> from emakhuwa_tokenizer import tokenize_sentence
>>> results = tokenize_sentence("Mwana anaavaha mwalimu ehuku")
>>> for r in results:
...     print(f"{r.surface_form}: {r.root} ({r.word_class})")
Mwana: ana (noun)
anaavaha: vah (verb)
mwalimu: alimu (noun)
ehuku: huku (noun)
```

Example output shape:

```python
{
    "surface_form": "kinaavaha",
    "normalized_form": "kinaavaha",
    "root": "vah",
    "word_class": "verb",
    "tam": "PRES/FUT",
    "morphemes": [
        {"form": "ki", "slot": "SM", "gloss": "1SG"},
        {"form": "naa", "slot": "TAM", "gloss": "PRES/FUT"},
        {"form": "vah", "slot": "ROOT", "gloss": "vah"},
        {"form": "a", "slot": "FV", "gloss": "FV.IND/IMP/INF"}
    ]
}
```

## CLI usage

The package installs two equivalent commands:

```bash
emakhuwa-tokenize kinaavaha
makhuwa-tokenize kinaavaha
```

Text output:

```text
Word: kinaavaha
Class: verb
Root: vah
Lemma: ovaha
Morphemes:
  ki-     [SM    ] 1SG
  naa-    [TAM   ] PRES/FUT
  vah-    [ROOT  ] give
  -a      [FV    ] indicative
Confidence: 0.95
Path: rule
```

Analyze a sentence:

```bash
emakhuwa-tokenize "Mwana anaavaha mwalimu ehuku"
```

Only lemmas:

```bash
emakhuwa-tokenize --lemma kinaavaha
```

JSON output:

```bash
emakhuwa-tokenize --json kinaavaha nkivahale
```

For one word, JSON output is a single object. For multiple words or sentence input, it is an array of objects.

```json
{
  "word": "kinaavaha",
  "word_class": "verb",
  "root": "vah",
  "lemma": "ovaha",
  "morphemes": [
    {"form": "ki", "slot": "SM", "gloss": "1SG"},
    {"form": "naa", "slot": "TAM", "gloss": "PRES/FUT"},
    {"form": "vah", "slot": "ROOT", "gloss": "give"},
    {"form": "a", "slot": "FV", "gloss": "indicative"}
  ],
  "confidence": 0.95,
  "path": "rule"
}
```

Verbose analysis:

```bash
emakhuwa-tokenize --verbose kivanje
```

Process a file, one word per line:

```bash
emakhuwa-tokenize --file words.txt --json
```

Use a custom dictionary/training data directory:

```bash
emakhuwa-tokenize --data-dir data/ kinaavaha
```

## API reference

### `tokenize(word: str) -> MorphToken`

Tokenize a single Emakhuwa word using the packaged rule system.

```python
from emakhuwa_tokenizer import tokenize
analysis = tokenize("kivanje")
```

### `load_tokenizer(data_dir: str | None = None) -> Tokenizer`

Create a reusable tokenizer. If `data_dir` is supplied, the tokenizer can build a training lexicon from that directory. If omitted, it uses packaged JSON rules only.

```python
from emakhuwa_tokenizer import load_tokenizer

tok = load_tokenizer()
analysis = tok.analyze_word("nkivahale")
```

### `lemmatize(word: str) -> str`

Return a citation-form lemma. Verbs are returned as infinitives (`o- + root + -a`); nouns return the identified bare stem/root.

```python
from emakhuwa_tokenizer import lemmatize
print(lemmatize("kinaavaha"))  # ovaha
```

### `tokenize_sentence(text: str) -> list[MorphToken]`

Split a sentence on whitespace/punctuation and analyze each word in order.

```python
from emakhuwa_tokenizer import tokenize_sentence
results = tokenize_sentence("Mwana anaavaha mwalimu ehuku")
```

### `MorphToken`

Dataclass representing one analyzed word.

Important fields:

- `surface_form`
- `normalized_form`
- `morphemes`
- `root`
- `word_class`
- `noun_class`
- `tam`
- `confidence`
- `gloss`
- `diagnostics`

Use `to_dict()` for JSON serialization.

### `Morpheme`

Dataclass representing one morpheme.

Important fields:

- `form`
- `slot`
- `gloss`
- `position`
- `underlying`
- `confidence`

## Architecture overview

```text
raw word
  ↓
spelling normalization
  ↓
exact dictionary lookup, optional
  ↓
rule-based segmentation
  ├─ noun-class and relational-prefix analysis
  ├─ verb-template parser: NEG-SM-TAM-OP-ROOT-EXT-FV
  ├─ extension and vowel-harmony handling
  ├─ imbrication reversal
  └─ morphophonological candidate handling
  ↓
decision routing and confidence scoring
  ↓
MorphToken output
```

Rules are externalized as JSON under `emakhuwa_tokenizer/rule_data/`:

- `morphophonology.json`
- `verb_template.json`
- `noun_classes.json`
- `extensions.json`
- `imbrication.json`
- `negation.json`
- `spelling_normalization.json`

## Evaluation

The current external gold-standard benchmark contains 272 manually annotated items.

Final Phase 8/9 metrics on `evaluation/gold_standard.json`:

| Metric | Score |
|---|---:|
| Boundary precision | 0.7938 |
| Boundary recall | 0.8110 |
| Boundary F1 | 0.8023 |
| Lemma/root accuracy | 0.8566 |
| Word-class accuracy | 0.7647 |

Test suite:

```text
pytest tests/ -v
3 passed
```

## Development

Run tests:

```bash
pytest
```

Run external gold evaluation:

```bash
python scripts/evaluate_gold.py
```

Build package:

```bash
python -m build
```

```bash
 cd C:\Users\t-fali\Pictures\mystuff\Consutorias\MAPA\emakhuwa-tokenizer
 pip install -e .
 emakhuwa-tokenize --json "kinohimeryakani"
 emakhuwa-tokenize "Mwana anaavaha mwalimu ehuku"
 emakhuwa-tokenize --lemma "kinaavaha"
 pytest tests/ -v
```


## Contributing

Contributions are welcome, especially:

- additional manually verified gold data
- dialect-specific spelling normalization rules
- improved finite-state morphology rules
- better handling of lexicalized nouns/adjectives
- documentation of Emakhuwa morpheme inventories

Please keep linguistic rules transparent and prefer JSON-rule updates over opaque model behavior when possible.

## License

MIT License. See `LICENSE`.
