Metadata-Version: 2.4
Name: Tan_language
Version: 0.1.0
Summary: Tanzania Local Language (TLL) Python library for Kiswahili and Kisukuma.
Author: TLL Contributors
License: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Provides-Extra: local-llm
Requires-Dist: huggingface-hub>=0.27; extra == "local-llm"
Requires-Dist: llama-cpp-python>=0.3.4; extra == "local-llm"
Dynamic: license-file

# TLL — Tanzania Local Language

TLL is a Python library ecosystem for Tanzanian local-language technology. The project starts with Kiswahili (`sw`) and Kisukuma (`suk`) and is designed to support additional Tanzanian languages without rewriting the core architecture.

This repository currently focuses on the foundational pieces required for a serious, reusable library:

- language registry
- dataset provenance and validation
- raw/cleaned/reviewed/verified data pipeline
- dictionary-oriented data access
- translation API with conservative behavior
- text normalization and tokenization hooks
- CLI and documentation scaffolding

Important: some features are intentionally scaffolded and documented as future work rather than pretending they are fully implemented.

## Quick start

PowerShell:

```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -e .
```

To install the published package from PyPI:

```powershell
pip install Tan_language
```

The package includes the cleaned Sukuma dataset needed by the dictionary
translation API, so `LOCAL_LANGUAGE` and the repository's root `data/` folder
are not required after installation.

Then:

```python
from tll import get_language, list_languages

print(get_language("suk").name)
print([lang.code for lang in list_languages()])
```

## Translation and chatbot usage

The translation API now uses the dataset-first algorithm from the
`LOCAL_LANGUAGE` project:

1. exact match
2. word-by-word composition for phrases
3. fuzzy spelling match
4. explicit "not found" response instead of invented translations

PowerShell examples:

```powershell
tan-language translate --from suk --to sw mwalimu
tan-language translate --from suk --to sw "mwalimu shule"
tan-language chat --from suk --to sw mwalimu
```

Python usage:

```python
from tll import LocalLanguageChatbot, Translator

translator = Translator(source="suk", target="sw")
print(translator.translate("mwalimu shule"))

chatbot = LocalLanguageChatbot(language="suk")
print(chatbot.chat("mwalimu", source_language="suk", target_language="sw"))
```

The chatbot is grounded in the local dataset. `grounded=True` means the
response came from a dataset match; a `grounded=False` response must be
reviewed by a native speaker and should not be treated as a verified
translation.

### Optional offline LLM chatbot

For natural conversation, install the optional GGUF runtime:

```powershell
pip install "Tan_language[local-llm]"
```

The first LLM request downloads the small Qwen instruct GGUF model into the
Hugging Face cache. Later requests reuse that file and run locally without
network access. Model weights are intentionally not included in the wheel.
To pre-download or use a model file managed by your organization, pass its
local path:

```powershell
tll chat --llm --model-path .\models\assistant.gguf --from suk --to sw "Habari?"
```

Or use Python:

```python
from tll import LocalLanguageChatbot

bot = LocalLanguageChatbot(backend="llm")
print(bot.chat("Habari?", source_language="sw", target_language="sw")["response"])
bot.reset()  # clear conversation history; the model stays loaded
```

Generation is deterministic by default (`temperature=0`, fixed seed). The
LLM is not a verified translation engine: responses have `grounded=False` and
should be reviewed for language accuracy.

## Current validation workflow

```powershell
python scripts/validate_dataset.py --dataset data/raw/suk/sukuma_deduplicated.csv
```

This validates the provided Kisukuma dataset without modifying the original raw file.

## Reviewed and verified dataset workflow

```powershell
python scripts/review_dataset.py --input data/cleaned/suk/cleaned_sukuma_dataset.csv --output-dir data/reviewed/suk --language suk
```

This creates the reviewed stage and then promotes the verified subset into a dedicated verified dataset. The library keeps all intermediate data, so the raw source remains untouched and the review decisions remain auditable.

## Project status

This is Phase 1 of the roadmap: dataset inspection, architecture, language registry, and validation. The cleaning stage begins only after validation is complete and reviewed.

## License

This project is licensed under the MIT license. Dataset licenses are handled separately and must be recorded carefully; if unknown, the metadata must state `unknown`.
