Metadata-Version: 2.4
Name: meranaw-tokenizer
Version: 0.1.4
Summary: A Meranaw tokenizer library trained specifically on translated religious texts (Bible and Quran).
Project-URL: Homepage, https://github.com/MantokKa/meranaw-tokenizer
Project-URL: Bug Tracker, https://github.com/MantokKa/meranaw-tokenizer/issues
Author-email: "Jamail M. Cali" <cali.jm099@s.msumain.edu.ph>
License-File: LICENSE
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.7
Requires-Dist: sentencepiece>=0.1.99
Description-Content-Type: text/markdown

# Meranaw Tokenizer

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A Python library for tokenizing the Meranaw language using subword segmentation models (BPE and Unigram). These models were trained only on translated Meranaw religious texts (Quran and Bible). This library was developed as part of a thesis study to facilitate the development of Meranaw NLP applications.

## Features

- **Two Algorithms**: Supports Byte Pair Encoding (BPE) and Unigram Language Model.
- **Three Sizes**: Includes Small, Medium, and Large vocabulary models.
- **Simple API**: Easy encoding and decoding functions.

## Installation

```bash
pip install meranaw-tokenizer
```

## Quick Start

```python
import meranaw_tokenizer as mt

# 1. Load a model (Default: Large Unigram)
tokenizer = mt.load_model()

# 2. Tokenize text
text = "Mapiya kapipita" 
text_lower = text.lower() 
tokens = tokenizer.encode(text_lower, out_type=str)
ids = tokenizer.encode(text_lower, out_type=int)

print(f"Tokens: {tokens}")
print(f"IDs: {ids}")

# 3. Decode back to text
decoded_text = tokenizer.decode(ids)
print(f"Decoded: {decoded_text}")
```

## Available Model Sizes

| Size | Vocab Count |
|------|-------------|
| `small` | 2,315 |
| `medium` | 6,402 |
| `large` | 19,859 |

## Usage Notes

- **Normalization**: These models were trained on normalized, lowercase Meranaw text. To maintain low subword fertility (preventing unnecessary fragmentation), always `.lower()` your string before encoding.
- **Algorithms**:
    - **Unigram**: Uses a probabilistic approach to tokenization.
    - **BPE**: Uses a frequency-based merging approach.


## Author

**Jamail M. Cali**  
MSU Main - Marawi City  
Email: [cali.jm099@s.msumain.edu.ph](mailto:cali.jm099@s.msumain.edu.ph)

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
