Metadata-Version: 2.4
Name: sandesh-unicode
Version: 1.0.0
Summary: 1-byte encoding for Nepali text - 66% smaller than UTF-8
Author-email: Sandesh Bastola <sandesh@bastola.com.np>
License-Expression: MIT
Project-URL: Homepage, https://github.com/sandesh-unicode/sandesh-unicode
Project-URL: Source, https://github.com/sandesh-unicode/sandesh-unicode
Project-URL: Tracker, https://github.com/sandesh-unicode/sandesh-unicode/issues
Keywords: nepali,encoding,unicode,devanagari,compression
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Text Processing :: Filters
Classifier: Topic :: System :: Archiving :: Compression
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: pre-commit>=3.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Requires-Dist: pytest-cov>=4.0; extra == "test"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.5; extra == "docs"
Requires-Dist: mkdocs-material>=9.0; extra == "docs"
Dynamic: license-file

# सन्देश Unicode

[![CI](https://github.com/sandesh-unicode/sandesh-unicode/actions/workflows/ci.yml/badge.svg)](https://github.com/sandesh-unicode/sandesh-unicode/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/sandesh-unicode.svg)](https://pypi.org/project/sandesh-unicode/)
[![Python Versions](https://img.shields.io/pypi/pyversions/sandesh-unicode.svg)](https://pypi.org/project/sandesh-unicode/)
[![License](https://img.shields.io/pypi/l/sandesh-unicode.svg)](https://github.com/sandesh-unicode/sandesh-unicode/blob/main/LICENSE)
[![Code Coverage](https://img.shields.io/codecov/c/github/sandesh-unicode/sandesh-unicode)](https://codecov.io/gh/sandesh-unicode/sandesh-unicode)

**Efficient 1-Byte Encoding for Nepali Text** — सन्देश encodes Nepali Unicode characters as single bytes instead of the 3-byte UTF-8 representation, achieving **~66% size reduction** for pure Nepali text.

| Metric              | UTF-8      | सन्देश     | Savings |
|---------------------|------------|------------|---------|
| Pure Nepali         | 3 bytes/ch | 1 byte/ch  | ~66%    |
| Mixed Nepali+English| varies     | varies     | ~40-50% |
| ASCII only          | 1 byte/ch  | 1 byte/ch  | 0%      |

---

## Installation

```bash
pip install sandesh-unicode
```

For development:

```bash
pip install "sandesh-unicode[dev]"
```

---

## Quick Start

### CLI

```bash
# Encode Nepali text
sandesh encode "सन्देश"
# Output: b'\xaa\x9e\x9c\xb2\xa8'

# Decode bytes back to text
echo -ne '\xaa\x9e\x9c\xb2\xa8' | sandesh decode
# Output: सन्देश

# Show compression statistics
sandesh stats "सन्देश नेपाल"

# Encode/decode files
sandesh encode-file input.txt output.sndsh
sandesh decode-file output.sndsh decoded.txt

# View mapping table
sandesh table

# Package info
sandesh info
```

### Python API

```python
from sandesh_unicode import encode, decode, get_compression_ratio

# Basic encode/decode
encoded = encode("सन्देश")
print(encoded)           # b'\xaa\x9e\x9c\xb2\xa8'
decoded = decode(encoded)
print(decoded)           # सन्देश

# Mixed Nepali-English text
encoded = encode("नमस्ते नेपाल")
decoded = decode(encoded)
print(decoded)           # नमस्ते नेपाल

# Compression statistics
stats = get_compression_ratio("सन्देश")
print(f"Saved {stats['savings_percent']:.1f}%")  # Saved ~66.7%

# Permissive mode (skip unsupported characters)
encoded = encode("Hello सन्देश 中文", strict=False)

# File operations
from sandesh_unicode import encode_file, decode_file
encode_file("nepali.txt", "nepali.sndsh")
decode_file("nepali.sndsh", "nepali_decoded.txt")

# Validation
from sandesh_unicode import is_sandesh_encoded, validate
data = encode("सन्देश")
print(is_sandesh_encoded(data))   # True
errors = validate(data)
print(len(errors))                # 0
```

---

## Architecture

```
┌─────────────┐     encode()     ┌──────────────┐
│  Unicode     │ ──────────────→ │  1-byte code  │
│  (UTF-8)     │ ←────────────── │  (0x80-0xFF)  │
│  3 bytes/ch  │     decode()    │  1 byte/ch    │
└─────────────┘                  └──────────────┘
     │                                │
     │ UTF-8 preserved for ASCII      │ ASCII passthrough
     │ (0x00-0x7F)                    │ (0x00-0x7F)
     └────────────────────────────────┘
```

### Memory Layout

| Range    | Category         | Count |
|----------|------------------|-------|
| 0x00-0x7F| ASCII (passthrough) | 128 |
| 0x80-0x8A| Vowels (अ-औ)     | 11   |
| 0x8B-0xAB| Consonants (क-ह) | 33   |
| 0xAC-0xB6| Matras/Vowel signs | 11 |
| 0xB7-0xBA| Special signs     | 4   |
| 0xBB-0xBC| Punctuation       | 2   |
| 0xBE-0xC7| Nepali digits (०-९) | 10 |
| 0xC8-0xFF| Common conjuncts  | 56  |

---

## Use Cases

- **Database storage**: Store Nepali text using 66% less space
- **File compression**: Compress `.txt` files containing Nepali text
- **Network transfer**: Reduce bandwidth for Nepali text payloads
- **Embedded systems**: Efficient storage on memory-constrained devices
- **APIs**: Optimize Nepali text in REST/GraphQL payloads

See the [`examples/`](examples/) directory for backend (FastAPI) and frontend (JavaScript) implementations.

---

## Development

```bash
# Clone and install with dev dependencies
git clone https://github.com/sandesh-unicode/sandesh-unicode.git
cd sandesh-unicode
pip install -e ".[dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov=sandesh_unicode --cov-report=term

# Format code
black src/ tests/
isort --profile black src/ tests/

# Run pre-commit hooks
pre-commit run --all-files
```

### Project Structure

```
sandesh-unicode/
├── src/
│   └── sandesh_unicode/
│       ├── __init__.py      # Public API exports
│       ├── tables.py         # Character mapping tables
│       ├── encoder.py        # Encode Unicode → सन्देश bytes
│       ├── decoder.py        # Decode सन्देश bytes → Unicode
│       ├── exceptions.py     # Custom exception hierarchy
│       └── cli.py            # Command-line interface
├── tests/
│   └── test_sandesh.py       # Comprehensive test suite
├── examples/
│   ├── backend/              # FastAPI storage API example
│   └── frontend/             # Browser-based encoder/decoder
├── pyproject.toml            # Build & tool configuration
└── .github/workflows/ci.yml  # CI pipeline (3 OS × 6 Python versions)
```

---

## Limitations

- **Ecosystem-dependent**: Only works within systems using this library
- **Display-agnostic**: Must decode to Unicode for standard display
- **Not a Unicode replacement**: Internal compression scheme only
- **Nepali-only**: Currently supports Devanagari Nepali script

---

## License

MIT
