Metadata-Version: 2.4
Name: sandesh-unicode
Version: 1.0.1
Summary: 1-byte encoding for Nepali text - 66% smaller than UTF-8
Author-email: Sandesh Bastola <sandesh@bastola.com.np>
License: MIT License
        
        Copyright (c) 2024 Sandesh Bastola
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/sandesh-unicode/sandesh-unicode
Project-URL: Source, https://github.com/sandesh-unicode/sandesh-unicode
Project-URL: Tracker, https://github.com/sandesh-unicode/sandesh-unicode/issues
Keywords: nepali,encoding,unicode,devanagari,compression
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Text Processing :: Filters
Classifier: Topic :: System :: Archiving :: Compression
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: pre-commit>=3.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Requires-Dist: pytest-cov>=4.0; extra == "test"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.5; extra == "docs"
Requires-Dist: mkdocs-material>=9.0; extra == "docs"
Dynamic: license-file

# सन्देश Unicode

[![CI](https://github.com/sandesh-unicode/sandesh-unicode/actions/workflows/ci.yml/badge.svg)](https://github.com/sandesh-unicode/sandesh-unicode/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/sandesh-unicode.svg)](https://pypi.org/project/sandesh-unicode/)
[![Python Versions](https://img.shields.io/pypi/pyversions/sandesh-unicode.svg)](https://pypi.org/project/sandesh-unicode/)
[![License](https://img.shields.io/pypi/l/sandesh-unicode.svg)](https://github.com/sandesh-unicode/sandesh-unicode/blob/main/LICENSE)
[![Code Coverage](https://img.shields.io/codecov/c/github/sandesh-unicode/sandesh-unicode)](https://codecov.io/gh/sandesh-unicode/sandesh-unicode)

**Efficient 1-Byte Encoding for Nepali Text** — सन्देश encodes Nepali Unicode characters as single bytes instead of the 3-byte UTF-8 representation, achieving **~66% size reduction** for pure Nepali text.

| Metric              | UTF-8      | सन्देश     | Savings |
|---------------------|------------|------------|---------|
| Pure Nepali         | 3 bytes/ch | 1 byte/ch  | ~66%    |
| Mixed Nepali+English| varies     | varies     | ~40-50% |
| ASCII only          | 1 byte/ch  | 1 byte/ch  | 0%      |

---

## Installation

```bash
pip install sandesh-unicode
```

For development:

```bash
pip install "sandesh-unicode[dev]"
```

---

## Quick Start

### CLI

```bash
# Encode Nepali text
sandesh encode "सन्देश"
# Output: b'\xaa\x9e\x9c\xb2\xa8'

# Decode bytes back to text
echo -ne '\xaa\x9e\x9c\xb2\xa8' | sandesh decode
# Output: सन्देश

# Show compression statistics
sandesh stats "सन्देश नेपाल"

# Encode/decode files
sandesh encode-file input.txt output.sndsh
sandesh decode-file output.sndsh decoded.txt

# View mapping table
sandesh table

# Package info
sandesh info
```

### Python API

```python
from sandesh_unicode import encode, decode, get_compression_ratio

# Basic encode/decode
encoded = encode("सन्देश")
print(encoded)           # b'\xaa\x9e\x9c\xb2\xa8'
decoded = decode(encoded)
print(decoded)           # सन्देश

# Mixed Nepali-English text
encoded = encode("नमस्ते नेपाल")
decoded = decode(encoded)
print(decoded)           # नमस्ते नेपाल

# Compression statistics
stats = get_compression_ratio("सन्देश")
print(f"Saved {stats['savings_percent']:.1f}%")  # Saved ~66.7%

# Permissive mode (skip unsupported characters)
encoded = encode("Hello सन्देश 中文", strict=False)

# File operations
from sandesh_unicode import encode_file, decode_file
encode_file("nepali.txt", "nepali.sndsh")
decode_file("nepali.sndsh", "nepali_decoded.txt")

# Validation
from sandesh_unicode import is_sandesh_encoded, validate
data = encode("सन्देश")
print(is_sandesh_encoded(data))   # True
errors = validate(data)
print(len(errors))                # 0
```

---

## Architecture

```
┌─────────────┐     encode()     ┌──────────────┐
│  Unicode     │ ──────────────→ │  1-byte code  │
│  (UTF-8)     │ ←────────────── │  (0x80-0xFF)  │
│  3 bytes/ch  │     decode()    │  1 byte/ch    │
└─────────────┘                  └──────────────┘
     │                                │
     │ UTF-8 preserved for ASCII      │ ASCII passthrough
     │ (0x00-0x7F)                    │ (0x00-0x7F)
     └────────────────────────────────┘
```

### Memory Layout

| Range    | Category         | Count |
|----------|------------------|-------|
| 0x00-0x7F| ASCII (passthrough) | 128 |
| 0x80-0x8A| Vowels (अ-औ)     | 11   |
| 0x8B-0xAB| Consonants (क-ह) | 33   |
| 0xAC-0xB6| Matras/Vowel signs | 11 |
| 0xB7-0xBA| Special signs     | 4   |
| 0xBB-0xBC| Punctuation       | 2   |
| 0xBE-0xC7| Nepali digits (०-९) | 10 |
| 0xC8-0xFF| Common conjuncts  | 56  |

---

## Use Cases

- **Database storage**: Store Nepali text using 66% less space
- **File compression**: Compress `.txt` files containing Nepali text
- **Network transfer**: Reduce bandwidth for Nepali text payloads
- **Embedded systems**: Efficient storage on memory-constrained devices
- **APIs**: Optimize Nepali text in REST/GraphQL payloads

See the [`examples/`](examples/) directory for backend (FastAPI) and frontend (JavaScript) implementations.

---

## Development

```bash
# Clone and install with dev dependencies
git clone https://github.com/sandesh-unicode/sandesh-unicode.git
cd sandesh-unicode
pip install -e ".[dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov=sandesh_unicode --cov-report=term

# Format code
black src/ tests/
isort --profile black src/ tests/

# Run pre-commit hooks
pre-commit run --all-files
```

### Project Structure

```
sandesh-unicode/
├── src/
│   └── sandesh_unicode/
│       ├── __init__.py      # Public API exports
│       ├── tables.py         # Character mapping tables
│       ├── encoder.py        # Encode Unicode → सन्देश bytes
│       ├── decoder.py        # Decode सन्देश bytes → Unicode
│       ├── exceptions.py     # Custom exception hierarchy
│       └── cli.py            # Command-line interface
├── tests/
│   └── test_sandesh.py       # Comprehensive test suite
├── examples/
│   ├── backend/              # FastAPI storage API example
│   └── frontend/             # Browser-based encoder/decoder
├── pyproject.toml            # Build & tool configuration
└── .github/workflows/ci.yml  # CI pipeline (3 OS × 6 Python versions)
```

---

## Limitations

- **Ecosystem-dependent**: Only works within systems using this library
- **Display-agnostic**: Must decode to Unicode for standard display
- **Not a Unicode replacement**: Internal compression scheme only
- **Nepali-only**: Currently supports Devanagari Nepali script

---

## License

MIT
