Metadata-Version: 2.4
Name: shobdo
Version: 0.2.2
Summary: A modern, high-performance Python library for Bengali dictionary operations.
Author-email: Inan <inan@iseer.co>
License-Expression: Apache-2.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: Bengali
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: pydantic>=2.0.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.3; extra == 'dev'
Provides-Extra: semantic
Requires-Dist: numpy; extra == 'semantic'
Requires-Dist: sentence-transformers; extra == 'semantic'
Description-Content-Type: text/markdown

<div align="center">

<img src="https://raw.githubusercontent.com/InanXR/ProjectShobdo/main/assets/banner.png" alt="Shobdo" width="600"/>

# Shobdo

**A comprehensive Bengali dictionary and semantic search library for Python.**

[![PyPI](https://img.shields.io/pypi/v/shobdo.svg)](https://pypi.org/project/shobdo/)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://pypi.org/project/shobdo/)
[![License](https://img.shields.io/badge/license-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Code Style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

[Installation](#installation) · [Quick Start](#quick-start) · [API Reference](#api-reference) · [Semantic Search](#semantic-search) · [Contributing](#development)

</div>

---

Shobdo (শব্দ, "word") provides fast, type-safe access to a 45,630-word Bengali dictionary backed by SQLite and pre-computed neural embeddings. It is designed to replace slow, memory-heavy JSON-based dictionary loading with an indexed database that starts in milliseconds.

| | JSON Loading | Shobdo |
|---|---|---|
| **Startup** | ~1,200 ms | ~6 ms |
| **Memory** | ~180 MB | < 5 MB |
| **Return type** | `dict` | Pydantic `Word` |

## Features

- **Exact, partial, and fuzzy search** on Bengali headwords with configurable Levenshtein distance
- **Reverse lookup** from English translations to Bengali words
- **Semantic search** across all 45,630 entries using pre-computed 512-dimensional neural embeddings
- **Cross-lingual queries** — search in English, retrieve results in Bengali
- **Synonym discovery** via vector similarity, with no model inference at runtime
- **Type-safe models** — every result is a validated Pydantic `Word` object
- **Zero configuration** — all data ships with the package; just `pip install` and go

## Installation

```bash
pip install shobdo
```

To enable semantic search capabilities (concept search, synonym finding):

```bash
pip install "shobdo[semantic]"
```

> The `semantic` extra installs `numpy` and `sentence-transformers`. The base package requires only `pydantic`.

## Quick Start

```python
from shobdo import Shobdo

with Shobdo() as d:
    word = d.search("স্বাধীনতা")
    print(word.word)                # স্বাধীনতা
    print(word.pronunciation)       # শ্বাধীন্তা
    print(word.meanings)            # ['স্বাধীন হওয়ার ভাব', ...]
    print(word.english_translation) # Independence / Freedom
    print(word.part_of_speech)      # বিশেষ্য
```

`Shobdo` implements the context manager protocol. You can also instantiate it directly with `d = Shobdo()` and call `d.close()` when done.

## API Reference

All methods are available on the `Shobdo` class.

### `search(word) -> Optional[Word]`

Exact match lookup by Bengali headword. Returns `None` if not found.

```python
result = d.search("অ")
```

### `lookup(query) -> List[Word]`

Partial (substring) match on Bengali headwords.

```python
words = d.lookup("ঋণ")
for w in words:
    print(w.word)
# ঋণ, ঋণগ্রস্ত, ঋণদাতা, ঋণপত্র, অঋণ, ...
```

### `search_fuzzy(query, max_distance=2) -> List[Word]`

Fuzzy matching using Levenshtein edit distance. Useful for correcting user typos.

```python
matches = d.search_fuzzy("পাকি", max_distance=1)
# Returns: [Word(word='পাখি', ...)]  — corrects the typo
```

### `search_english(query) -> List[Word]`

Reverse lookup by English translation (substring match).

```python
words = d.search_english("freedom")
for w in words:
    print(w.word, "-", w.english_translation)
# স্বাধীনতা - Independence / Freedom
# মুক্তি - Freedom / Release
```

### `get_random() -> Word`

Returns a random dictionary entry.

```python
word = d.get_random()
print(f"{word.word}: {word.meanings[0]}")
```

### `search_by_pos(pos, limit=50) -> List[Word]`

Filter the dictionary by part of speech.

```python
nouns = d.search_by_pos("বিশেষ্য", limit=5)
for w in nouns:
    print(w.word, "-", w.english_translation)
```

### `search_many(words) -> Dict[str, Optional[Word]]`

Batch lookup of multiple words in a single query. Returns `None` for words not found.

```python
results = d.search_many(["অ", "আনন্দ", "nonexistent"])
for word, entry in results.items():
    print(word, "→", entry.english_translation if entry else "not found")
```

### `stats() -> Dict`

Returns a summary of the dictionary state.

```python
d.stats()
# {'total_words': 45630, 'backend': 'sqlite', 'semantic_search': True}
```

---

## Semantic Search

Shobdo ships with pre-computed 512-dimensional embeddings for every word in the dictionary. These vectors enable meaning-based retrieval: given a natural-language query, the library finds words that are _conceptually_ related, even when they share no common characters.

The `semantic` extra is required: `pip install "shobdo[semantic]"`.

### `search_semantic(query, top_k=10) -> List[Tuple[Word, float]]`

Encode a free-text query at runtime and rank all dictionary entries by cosine similarity.

```python
results = d.search_semantic("happiness and joy", top_k=5)

for word, score in results:
    print(f"{word.word}: {word.english_translation} ({score:.2f})")
# হর্ষোদয়: Rise of joy (19.86)
# আনন্দকন্দ: Root of joy (19.43)
# হর্ষাবিষ্ট: Overwhelmed with joy (19.26)
# সুখানুভব: Feeling of happiness (18.77)
# আনন্দ: Joy / Happiness (18.46)
```

Queries can be in English or Bengali. The model operates cross-lingually.

### `find_similar(word, top_k=10) -> List[Tuple[Word, float]]`

Find semantically similar words (synonyms) using pre-computed embeddings. Because both sides of the comparison are pre-indexed, this method requires **no model inference at runtime** — it is a pure NumPy dot product.

```python
synonyms = d.find_similar("আনন্দ", top_k=5)

for word, score in synonyms:
    print(f"{word.word}: {word.english_translation}")
# হর্ষোদয়: Rise of joy
# তোষ: Satisfaction / Pleasure
# আনন্দকন্দ: Root of joy
# হরষিত: Delighted / Joyful
# শর্ম: Happiness / Shelter
```

---

## Data Models

All search results are returned as Pydantic models.

```python
from shobdo.models import Word, Etymology
```

**`Word`**

| Field | Type | Description |
|---|---|---|
| `word` | `str` | Bengali headword |
| `pronunciation` | `Optional[str]` | Phonetic pronunciation |
| `part_of_speech` | `Optional[str]` | Grammatical category |
| `meanings` | `List[str]` | Definitions |
| `english_translation` | `Optional[str]` | English equivalent |
| `examples` | `List[str]` | Usage examples |
| `etymology` | `Optional[Etymology]` | Origin information |

**`Etymology`**

| Field | Type | Description |
|---|---|---|
| `source_language` | `Optional[str]` | Source language (e.g., সংস্কৃত, আরবি) |
| `derivation` | `Optional[str]` | Morphological breakdown |

---

## Architecture

Shobdo separates data preparation (build-time) from data consumption (runtime) to keep the installed package fast and self-contained.

**Build-time artifacts** (shipped with the package):

| File | Size | Contents |
|---|---|---|
| `dictionary.db` | ~22 MB | SQLite database with indexed columns on `word` and `english_translation` |
| `embeddings.npy` | ~45 MB | 45,630 x 512 float16 embedding matrix |
| `word_index.json` | ~3 MB | Maps vector indices to database row IDs |

**Runtime dependencies:**

- **Core** — SQLite (stdlib) + Pydantic. No external services, no network calls.
- **Semantic** — adds NumPy (dot-product similarity) and `sentence-transformers` (query encoding).

---

## Development

```bash
git clone https://github.com/inanXR/ProjectShobdo
cd library
pip install -e ".[semantic,dev]"

# Run the test suite
pytest tests/ -v

# Rebuild the SQLite database from source data
python scripts/build_db.py

# Regenerate embeddings (GPU recommended; see scripts/generate_embeddings_colab.ipynb)
python scripts/generate_embeddings.py
```

## Requirements

| Dependency | Required for |
|---|---|
| Python 3.9+ | All |
| `pydantic >= 2.0` | All |
| `numpy` | Semantic search |
| `sentence-transformers` | Semantic search (query encoding) |

## License

[Apache License 2.0](https://opensource.org/licenses/Apache-2.0)

---

<div align="center">

[GitHub](https://github.com/inanXR/ProjectShobdo) · [Issues](https://github.com/inanXR/ProjectShobdo/issues) · [PyPI](https://pypi.org/project/shobdo/)

Built by [Inan](https://github.com/inanXR).

</div>