Metadata-Version: 2.4
Name: bengali_bpe
Version: 0.1.2
Summary: A Byte Pair Encoding (BPE) library for the Bengali language.
Home-page: https://github.com/yourusername/bengali_bpe
Author: Firoj Ahmmed Patwary
Author-email: your.email@example.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Natural Language :: Bengali
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python
Dynamic: summary

# Bengali BPE

**`bengali_bpe`** is a Python library for **Byte Pair Encoding (BPE)** specifically designed for the **Bengali language**.  
It enables you to train BPE models on Bengali text, encode words and sentences into subword units, and decode them back.  
This helps improve NLP model performance for Bengali text processing, tokenization, and embedding preparation.

---

## ✨ Features

- 🧠 Train a Byte Pair Encoding model on Bengali text corpus  
- 🔠 Encode Bengali sentences or words into subword tokens  
- 🔁 Decode subword tokens back into full Bengali words  
- ⚙️ Simple, lightweight, and easy to integrate into your NLP pipelines  
- 🪶 Supports Bengali Unicode normalization

---

## 📦 Installation

Install directly from PyPI:

```bash
pip install bengali_bpe

from bengali_bpe import BengaliBPE
from bengali_bpe.utils import normalize_bengali_text

# Sample Bengali corpus
corpus = [
    "বাংলা ভাষা সুন্দর",
    "আমি বাংলা পড়ি",
    "বাংলা ভয়ানক নয়"
]

# Normalize text
corpus = [normalize_bengali_text(sentence) for sentence in corpus]

# Initialize and train the model
bpe = BengaliBPE(num_merges=10)
bpe.train(corpus)

# Encode a sentence
sentence = "বাংলা ভাষা সুন্দর"
encoded = bpe.encode(sentence)
print("Encoded:", encoded)

# Decode back
decoded = bpe.decode(encoded)
print("Decoded:", decoded)


Encoded: [['বা', 'ংলা'], ['ভা', 'ষা'], ['সু', 'ন্', 'দর']]
Decoded: বাংলা ভাষা সুন্দর


from bengali_bpe import BengaliBPE

bpe = BengaliBPE(num_merges=5)
bpe.train(["বাংলা ভাষা সুন্দর"])
encoded_word = bpe.encode_word("বাংলা")
print("Encoded Word:", encoded_word)

decoded_word = bpe.decode([encoded_word])
print("Decoded Word:", decoded_word)


Encoded Word: ['বা', 'ংলা']
Decoded Word: বাংলা




