Metadata-Version: 2.4
Name: swahiliclean
Version: 0.1.0
Summary: A comprehensive Python package for preprocessing Swahili (Kiswahili) text data. Provides functions for text normalization, stopword removal, slang normalization, typo correction, and tokenization. Includes curated datasets of Swahili stopwords, slang terms, and common typos. Designed for NLP tasks on Swahili text corpora.
Author-email: Bernard Masua <bhrmasua@gmail.com>, Noel Masasi <noeliasmasasi@gmail.com>
License: MIT
Project-URL: Homepage, https://example.com/swahiliclean
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest>=7.4; extra == "test"
Dynamic: license-file

# swahiliclean (Python)

Python port of the Swahili text preprocessing toolkit for NLP on Kiswahili text.

## Quick Start

### Installation

```bash
pip install -e .
```

For development with tests:
```bash
pip install -e .[test]
python3 -m pytest
```

### Basic Usage

```python
from swahiliclean import preprocess_sw, tokenize_sw_words

# Full preprocessing pipeline
text = "manzi poa sana 2na tufanye kazi na sisi"
cleaned = preprocess_sw(text)
print(cleaned)

# Tokenization
tokens = tokenize_sw_words(text)
print(tokens)  # ["manzi", "poa", "sana", ...]
```

### Batch Processing

All functions support batch processing with lists:

```python
sentences = [
    "manzi poa sana 2na tufanye kazi na sisi",
    "Habari za asubuhi? Nimekuja kwa ajili ya kazi.",
    "Na sisi tunaenda kwa shule leo."
]

# Process all at once
processed = preprocess_sw(sentences)
for original, cleaned in zip(sentences, processed):
    print(f"{original} -> {cleaned}")
```

## Features

- **Text Normalization**: Lowercase, trim, collapse whitespace
- **Stopword Removal**: Remove common Swahili stopwords (~258 words)
- **Slang Normalization**: Replace slang with standard forms (~190 mappings)
- **Typo Correction**: Fix common typos (~422 corrections)
- **Tokenization**: Simple word tokenization
- **Email & URL Removal**: Detect and remove email addresses and URLs
- **Text Statistics**: Count and analyze stopwords, slang, typos with percentages
- **Full Pipeline**: Combine all steps with configurable options

## Available Functions

### Core Functions
- `normalize_sw_text()` - Text normalization
- `remove_sw_stopwords()` - Stopword removal
- `normalize_sw_slang()` - Slang normalization
- `correct_sw_typos()` - Typo correction
- `tokenize_sw_words()` - Word tokenization
- `preprocess_sw()` - Full preprocessing pipeline

### Email & URL Removal
- `remove_emails()` - Remove email addresses
- `remove_urls()` - Remove URLs
- `remove_emails_and_urls()` - Remove both emails and URLs

### Text Statistics
- `count_sw_words()` - Count words in text
- `count_sw_stopwords()` - Count stopwords found
- `count_sw_slang()` - Count slang terms found
- `count_sw_typos()` - Count typos found
- `get_text_statistics()` - Get comprehensive statistics
- `get_cleaning_statistics()` - Compare before/after cleaning

## Custom Data

You can provide your own stopwords, typos, or slang mappings:

```python
from swahiliclean import preprocess_sw

# Custom stopwords
result = preprocess_sw(
    "na sisi tunaenda yangu",
    custom_stopwords=["yangu", "yako"]
)

# Custom typos
result = preprocess_sw(
    "kwahiyo tunaenda",
    custom_typos=[("kwahiyo", "kwa hiyo")]
)

# Custom slang
result = preprocess_sw(
    "hii kazi ni mzito",
    custom_slang=[("mzito", "mzito sana")]
)
```

## Configuration Options

All functions support extensive configuration:

- **Case sensitivity**: Control case-sensitive matching
- **Word boundaries**: Toggle word boundary matching for stopwords
- **Case preservation**: Preserve original case when needed
- **Whitespace handling**: Preserve newlines/tabs or normalize them
- **Custom patterns**: Use custom regex patterns for tokenization
- **Step control**: Enable/disable individual pipeline steps

## Documentation

For complete documentation including:
- Full parameter reference for all functions
- Detailed examples and use cases
- Edge cases and behavior notes
- Advanced configuration options

See **[DOCUMENTATION.md](DOCUMENTATION.md)** for the complete manual.

## Data

Bundled data files (CSV format):
- `swahiliclean/data/Stopwords.csv` - ~258 stopwords
- `swahiliclean/data/Slangs.csv` - ~190 slang mappings
- `swahiliclean/data/Typos.csv` - ~422 typo corrections

## Examples

### Step-by-Step Processing

```python
from swahiliclean import (
    normalize_sw_text,
    remove_sw_stopwords,
    correct_sw_typos,
    normalize_sw_slang
)

text = "manzi poa sana 2na tufanye kazi na sisi"

# Step 1: Normalize
normalized = normalize_sw_text(text)

# Step 2: Correct typos
corrected = correct_sw_typos(normalized)

# Step 3: Normalize slang
slang_normalized = normalize_sw_slang(corrected)

# Step 4: Remove stopwords
final = remove_sw_stopwords(slang_normalized)
```

### Advanced Configuration

```python
# Preserve case and newlines
result = preprocess_sw(
    "Habari YAKO\nleo",
    lowercase=False,
    preserve_newlines=True,
    remove_stop=False
)

# Case-sensitive processing
result = preprocess_sw(
    "Na sisi",
    stopwords_case_sensitive=True,
    slang_case_sensitive=True
)

# Remove emails and URLs
result = preprocess_sw(
    "Contact test@example.com or visit https://site.com",
    remove_emails_flag=True,
    remove_urls_flag=True
)
```

### Text Statistics

```python
from swahiliclean import get_text_statistics, get_cleaning_statistics, preprocess_sw

# Get statistics before cleaning
text = "na sisi tunaenda kwa shule"
stats = get_text_statistics(text)
print(f"Stopwords: {stats['stopwords_count']} ({stats['stopwords_percentage']}%)")
print(f"Total detected: {stats['total_detected']} ({stats['total_percentage']}%)")

# Compare before and after
before = "na sisi tunaenda kwa shule"
after = preprocess_sw(before)
cleaning_stats = get_cleaning_statistics(before, after)
print(f"Words before: {cleaning_stats['words_before']}")
print(f"Words after: {cleaning_stats['words_after']}")
print(f"Words removed: {cleaning_stats['words_removed']} ({cleaning_stats['words_removed_percentage']}%)")
```

## Testing

Run the test suite:
```bash
python3 -m pytest
```

Quick batch testing:
```bash
python3 tests/batch_demo.py
```

## Authors

- Bernard Masua <bhrmasua@gmail.com>
- Noel Masasi <noeliasmasasi@gmail.com>

## License

MIT License (see `LICENSE`).
