Metadata-Version: 2.4
Name: swahiliclean
Version: 0.3.0
Summary: A comprehensive Python package for preprocessing Swahili (Kiswahili) text data. Provides functions for text normalization, stopword removal, slang normalization, typo correction, and tokenization. Includes curated datasets of Swahili stopwords, slang terms, and common typos. Designed for NLP tasks on Swahili text corpora.
Author-email: Bernard Masua <bhrmasua@gmail.com>, Noel Masasi <noeliasmasasi@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://pypi.org/project/swahiliclean/
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest>=7.4; extra == "test"
Dynamic: license-file

# swahiliclean (Python)

Python port of the Swahili text preprocessing toolkit for NLP on Kiswahili text.

## Quick Start

### Installation

```bash
pip install swahiliclean
```

For development with tests:
```bash
pip install -e .[test]
python3 -m pytest
```

### Basic Usage

```python
from swahiliclean import preprocess_sw, tokenize_sw_words

# Full preprocessing pipeline
text = "manzi poa sana 2na tufanye kazi na sisi"
cleaned = preprocess_sw(text)
print(cleaned)

# Tokenization
tokens = tokenize_sw_words(text)
print(tokens)  # ["manzi", "poa", "sana", ...]
```

### Batch Processing

All functions accept a list of strings. Files and directories have their own helpers:

```python
sentences = [
    "manzi poa sana 2na tufanye kazi na sisi",
    "Habari za asubuhi? Nimekuja kwa ajili ya kazi.",
    "Na sisi tunaenda kwa shule leo."
]

# Process all at once
processed = preprocess_sw(sentences)
for original, cleaned in zip(sentences, processed):
    print(f"{original} -> {cleaned}")

# One file, or a directory of .txt files
from swahiliclean import preprocess_sw_file, preprocess_sw_corpus

preprocess_sw_file("notes.txt", output_path="notes.clean.txt", by_line=True)
cleaned_docs = preprocess_sw_corpus("corpus/")
```

## Features

- **Text Normalization**: Lowercase, trim, collapse whitespace
- **Stopword Removal**: Remove common Swahili stopwords (~258 words)
- **Slang Normalization**: Replace slang with standard forms (~190 mappings)
- **Typo Correction**: Fix common typos (~422 corrections)
- **Tokenization**: Simple word tokenization
- **Email & URL Removal**: Detect and remove email addresses and URLs
- **Text Statistics**: Count and analyze stopwords, slang, typos with percentages
- **Full Pipeline**: Combine all steps with configurable options

## Available Functions

### Core Functions
- `normalize_text()` - Text normalization
- `remove_stopwords()` - Stopword removal
- `normalize_slang()` - Slang normalization
- `correct_typos()` - Typo correction
- `tokenize_words()` - Word tokenization
- `preprocess_sw()` - Full preprocessing pipeline

### Aliases
The short names above are the same functions as:
- `normalize_sw_text()`
- `remove_sw_stopwords()`
- `normalize_sw_slang()`
- `correct_sw_typos()`
- `tokenize_sw_words()`

### Batch
- `preprocess_sw_file()` - Clean one text file
- `preprocess_sw_corpus()` - Clean a list, a line-delimited file, or a directory of `.txt` files

### Data
- `prepare_swahili_data()` - Load stopwords, slang, and typos, including optional files and custom entries
- `load_stopwords()`, `load_slang()`, `load_typos()` - Load one bundled or user dictionary

### Email & URL Removal
- `remove_emails()` - Remove email addresses
- `remove_urls()` - Remove URLs
- `remove_emails_and_urls()` - Remove both emails and URLs
- `remove_non_alphanumeric()` - Remove non-alphanumeric characters
- `remove_non_alphabetic()` - Remove non-alphabetic characters
- `remove_short_sw_words()` - Remove short words by minimum length
- `remove_emojis()` - Remove emoji characters
- `remove_unicode_characters()` - Remove non-ASCII Unicode characters

### Text Statistics
- `count_sw_words()` - Count words in text
- `count_sw_stopwords()` - Count stopwords found
- `count_sw_slang()` - Count slang terms found
- `count_sw_typos()` - Count typos found
- `get_text_statistics()` - Get comprehensive statistics
- `get_cleaning_statistics()` - Compare before/after cleaning

## Custom Data

You can provide your own stopwords, typos, or slang mappings:

```python
from swahiliclean import preprocess_sw, prepare_swahili_data

# Custom stopwords
result = preprocess_sw(
    "na sisi tunaenda yangu",
    custom_stopwords=["yangu", "yako"]
)

# Custom typos
result = preprocess_sw(
    "kwahiyo tunaenda",
    custom_typos=[("kwahiyo", "kwa hiyo")]
)

# Custom slang
result = preprocess_sw(
    "hii kazi ni mzito",
    custom_slang=[("mzito", "mzito sana")]
)

# Replace a bundled dictionary with a file. Custom entries still merge on top.
result = preprocess_sw(
    "na zzqxstop tunaenda",
    stopwords_path="my_stopwords.txt",
    custom_stopwords=["tunaenda"]
)

data = prepare_swahili_data(stopwords_path="my_stopwords.txt")
```

## Configuration Options

| Option | Used in | Description |
|--------|---------|-------------|
| `ignore_case` | stopwords, slang, typos, pipeline, stats | Case-insensitive dictionary matching. `case_sensitive` still works |
| `word_boundaries` | stopwords, slang, typos, pipeline, stats | Whole-word matching versus substring matching |
| `to_lower` | normalize, pipeline | Lowercase text. `lowercase` still works |
| `preserve_newlines` / `preserve_tabs` | normalize, pipeline | Keep structure while collapsing spaces |
| `custom_stopwords` / `custom_slang` / `custom_typos` | core, pipeline, stats | Merge user dictionaries on top of the active list |
| `stopwords_path` / `slang_path` / `typos_path` | core, pipeline, stats | Load that dictionary from a CSV or text file instead of the bundled one |

Other controls:

- **Step control**: Enable or disable individual pipeline steps
- **Character filtering**: Remove non-alphanumeric or non-alphabetic characters, emojis, and non-ASCII characters
- **Word length filtering**: Remove short words using `min_word_length`
- **Custom patterns**: Use a custom regex for tokenization

## Documentation

For complete documentation including:
- Full parameter reference for all functions
- Detailed examples and use cases
- Edge cases and behavior notes
- Advanced configuration options

See **[DOCUMENTATION.md](DOCUMENTATION.md)** for the complete manual.

## Data

Bundled data files (CSV format):
- `swahiliclean/data/Stopwords.csv` - ~258 stopwords
- `swahiliclean/data/Slangs.csv` - ~190 slang mappings
- `swahiliclean/data/Typos.csv` - ~422 typo corrections

User dictionary files can be CSV or plain text. Stopwords are one word per line, or a CSV with a `StopWords` column. Slang and typos are `source,target` lines, or CSV columns such as `Slang,Meaning` and `Typo,Word`. Text pairs may also be separated with a tab, `|`, or `=>`.

## Examples

### Step-by-Step Processing

```python
from swahiliclean import (
    normalize_sw_text,
    remove_sw_stopwords,
    correct_sw_typos,
    normalize_sw_slang
)

text = "manzi poa sana 2na tufanye kazi na sisi"

# Step 1: Normalize
normalized = normalize_sw_text(text)

# Step 2: Correct typos
corrected = correct_sw_typos(normalized)

# Step 3: Normalize slang
slang_normalized = normalize_sw_slang(corrected)

# Step 4: Remove stopwords
final = remove_sw_stopwords(slang_normalized)
```

### Advanced Configuration

```python
# Preserve case and newlines
result = preprocess_sw(
    "Habari YAKO\nleo",
    to_lower=False,
    preserve_newlines=True,
    remove_stop=False
)

# Case-sensitive dictionary matching
result = preprocess_sw(
    "Na sisi",
    ignore_case=False,
    to_lower=False
)

# Remove emails and URLs
result = preprocess_sw(
    "Contact test@example.com or visit https://site.com",
    remove_emails_flag=True,
    remove_urls_flag=True
)
```

### Text Statistics

```python
from swahiliclean import get_text_statistics, get_cleaning_statistics, preprocess_sw

# Get statistics before cleaning
text = "na sisi tunaenda kwa shule"
stats = get_text_statistics(text)
print(f"Stopwords: {stats['stopwords_count']} ({stats['stopwords_percentage']}%)")
print(f"Total detected: {stats['total_detected']} ({stats['total_percentage']}%)")

# Compare before and after
before = "na sisi tunaenda kwa shule"
after = preprocess_sw(before)
cleaning_stats = get_cleaning_statistics(before, after)
print(f"Words before: {cleaning_stats['words_before']}")
print(f"Words after: {cleaning_stats['words_after']}")
print(f"Words removed: {cleaning_stats['words_removed']} ({cleaning_stats['words_removed_percentage']}%)")
```

## Testing

Run the test suite:
```bash
python3 -m pytest
```

Quick batch testing:
```bash
python3 tests/batch_demo.py
```

## Authors

- Bernard Masua <bhrmasua@gmail.com>
- Noel Masasi <noeliasmasasi@gmail.com>

## License

MIT License (see `LICENSE`).
