Metadata-Version: 2.4
Name: syntex-protocol
Version: 1.2.4
Summary: Three-layer semantic compression for LLM prompts (L1+L2+L3 + Lean Mode + Columnar JSON)
Project-URL: Homepage, https://github.com/syntex-dev/syntex
Project-URL: Repository, https://github.com/syntex-dev/syntex
Project-URL: Issues, https://github.com/syntex-dev/syntex/issues
Author-email: SyntEx Team <team@syntex.dev>
License-File: LICENSE
License-File: LICENSE.md
Keywords: compiler,compression,ir,llm,optimization,prompt
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Compilers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: requests>=2.28.0
Requires-Dist: tiktoken>=0.7.0
Requires-Dist: typer>=0.9.0
Provides-Extra: dev
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Description-Content-Type: text/markdown

# SyntEx

<p align="center">

[![PyPI](https://img.shields.io/pypi/v/syntex?color=2560bd)](https://pypi.org/project/syntex/)
[![Python](https://img.shields.io/pypi/pyversions/syntex?color=2560bd)](https://pypi.org/project/syntex/)
[![License](https://img.shields.io/pypi/l/syntex?color=green)](LICENSE)
[![Tests](https://img.shields.io/github/actions/workflow/status/syntex-dev/syntex/tests.yml?branch=main)](https://github.com/syntex-dev/syntex/actions)

</p>

> **Semantic Prompt Compression Protocol** — Three-layer compression for LLM system prompts.

---

## Why SyntEx?

| Feature | SyntEx | Plain Text | Savings |
|---------|--------|------------|---------|
| Multi-agent prompts (3+ agents) | 35-40 tokens | 100+ tokens | **60-71%** |
| RAG context chunks | 50-60% | 100% | **50-60%** |
| System prompts (repetitive boilerplate) | 50-60% | 100% | **50-60%** |
| JSON-heavy payloads | 36-40% | 100% | **60-68%** |

**Target: 50% token reduction** — SyntEx v1.2 achieves **60-68%** on typical multi-agent and JSON use cases.

### Benchmark Results (v1.2.0)

| Corpus Type | Original | Compressed | Reduction |
|-------------|----------|------------|-----------|
| Standard (repetitive) | 4,654 tokens | 2,405 tokens | 48.3% |
| + lorem ipsum | 10,468 tokens | 3,406 tokens | 67.5% |
| JSON-heavy (30 msgs) | 2,100 tokens | 764 tokens | **63.6%** |
| Multi-agent boilerplate | 3,730 tokens | 1,964 tokens | 47.3% |
| Unique content (worst) | 1,847 tokens | 1,626 tokens | 12.0% |

**Average: 46.4%** | **Peak: 67.5%**

---

## Configuration

SyntEx v1.2+ is fully configurable via `syntex.yaml`:

```yaml
# thresholds.yaml
version: "1.2.0"

thresholds:
  skip_cft: 25        # Text shorter triggers SKIP mode
  l1_cft: 80          # L1-only vs L1+L2 boundary
  min_l2_tokens: 30   # Minimum tokens for L2 compression

mirror:
  ttl_minutes: 1440   # Entry TTL (24 hours)
  max_capacity: 500   # Max mirror entries

pipeline:
  auto_minify: true   # Auto-minify JSON, Markdown, etc.
  two_pass_compression: true
  use_clustering: false
```

---

## Quick Start

```bash
pip install syntex
```

### Python API

```python
from syntex import SintExSession

session = SintExSession()
result = session.compile("Your long system prompt here...")
print(f"Compressed: {result.compressed_tokens} tokens (was {result.original_tokens})")
```

### CLI

```bash
python -m syntex compile myprompt.sx -o output.sxc
python -m syntex bench myprompt.sx
```

---

## Architecture

```text
┌─────────────────────────────────────────────────────────────────────────┐
│                             SyntEx Pipeline                             │
├─────────────────────────────────────────────────────────────────────────┤
│ L1: Global Vocabulary   →  72 Unicode symbols      (Greek + Braille)    │
│ L2: Local Dictionary    →  N-gram compression      (corpus-specific)    │
│ L3: Clustering          →  Semantic normalization  (variant merge)      │
└─────────────────────────────────────────────────────────────────────────┘
```

### Core Components

| Module | Purpose |
|--------|---------|
| `SintExSession` | Main API for compile/decompile |
| `SintExMirror` | Shared dictionary for multi-agent communication |
| `AutoVocab` | ROI-based vocabulary auto-promotion |
| `SintExGateway` | Proxy LLM with automatic compression |
| `Telemetry` | Track token savings and cost ($) |

---

## Features

### Three-Layer Compression

```python
from syntex import SintExSession

session = SintExSession(seed=["domain-specific term"])

# L1 + L2 + L3 compression
result = session.compile(system_prompt)

# Decompress
original = session.decompile(result.sxc)
```

### Lean Mode (No Header)

```python
# For multi-agent: send only compressed body, no header overhead
lean = session.compile_lean(system_prompt)
# Returns: body only, 2-3 token savings vs full format
```

### Multi-Agent Communication (SintExMirror)

```python
from syntex import SintExMirror

mirror = SintExMirror()

# Agent 1: Add candidates, build shared dictionary
candidates = ["Act as a senior architect", "Design scalable systems"]
mirror.add_candidates(candidates)
mirror.build_dict()

# Agent 2: Use same dictionary
compressed = mirror.compress(agent1_prompt, lean_mode=True)
decompressed = mirror.decompress(compressed)
```

### AutoVocab (Automatic ROI-Based Promotion)

```python
from syntex import AutoVocab

av = AutoVocab(threshold_roi=2.0)
av.feed(agent_interactions)
av.promote_top(max_promotions=5)
# Auto-promotes phrases with best token savings
```

### Gateway (LLM Proxy)

```python
from syntex import SintExGateway

gateway = SintExGateway(
    api_key="sk-...",
    model="gpt-4",
    base_url="https://api.openai.com/v1"
)

# Compression automatic
response = gateway.chat.completions.create(
    messages=[{"role": "system", "content": long_prompt}]
)
```

### Telemetry

```python
from syntex import Telemetry

t = Telemetry()
t.track("prompt_1", original_tokens=500, compressed_tokens=200)
t.report()
# Shows: savings %, dollars saved, ROI
```

---

## CLI

```bash
syntex compile <file.sx>      # Compile .sx → .sxc
syntex decompile <file.sxc>   # Decompile → text
syntex bench <file.sx>        # Benchmark compression
syntex tokens <file>          # Count tokens
syntex version                # Show version
```

---

## Format

### Input `.sx`
```text
$role=Act as a senior software architect

@system_architect
- $role
- Design scalable systems
- Consider trade-offs

@system_reviewer
- $role
- Review designs critically
```

### Output `.sxc`
```text
[SXC]
[DICT]
$a=Act as a senior software architect
[/DICT]
[BODY]
@system_architect
- $a
- Design scalable systems
...
[/BODY]
[/SXC]
```

### Lean Mode (no header)
```text
[DICT]
$a=Act as a senior software architect
[/DICT]
[BODY]
@system_architect
- $a
...
[/BODY]
```

---

## Changelog

### v1.2.0 (current)
- **Config-Driven**: All magic numbers extracted to `syntex.yaml`
- **Columnar JSON**: Automatic transformation of repetitive JSON to columnar format
- **Data Optimizations**: Integer scaling, delta timestamps, enum encoding
- **JSON Compression**: 0.2% → 63.6% on JSON-heavy payloads
- **Config Tests**: Full test coverage for configuration system

### v1.1.0
- **SintExMirror**: Multi-agent shared dictionary with consensus
- **Lean Mode**: Compression without header (2-3 token savings)
- **AutoVocab**: ROI-based automatic vocabulary promotion
- **Thread-Safety**: Locking and snapshot for concurrent ops
- **Graceful Fallbacks**: Auto-recovery on decompression failure

### v1.0
- Full L1+L2+L3 pipeline
- 72 global vocabulary entries (Greek + Braille)
- Telemetry tracking

### v0.9.x
- Initial releases with dual-layer compression

---

## Enterprise: Solve the Rate Limit Problem

**The Problem**: Every LLM provider (OpenAI, Anthropic, Google) has strict rate limits. Companies running multi-agent systems burn through tokens faster than they can scale.

**The Solution**: SyntEx reduces token usage by **50-70%** automatically.

| Metric | Before SyntEx | After SyntEx |
|--------|---------------|--------------|
| 10-agents prompts | ~8,000 tokens | ~3,200 tokens |
| API calls/day (budget) | 50,000 | 20,000 |
| Monthly cost (est. $10/1M) | $500 | $200 |
| Rate limit risk | High | **Eliminated** |

### Enterprise Features
- **SintExGateway**: Drop-in proxy that compresses automatically
- **Telemetry**: Track savings in dollars, not just tokens
- **SLA Guarantee**: Zero data loss (100% round-trip integrity)
- **On-premise option**: Your data never leaves your infrastructure

### Quick Integration
```python
from syntex import SintExGateway

gateway = SintExGateway(api_key="sk-...")
gateway.chat.completions.create(
    messages=[{"role": "system", "content": large_prompt}]
)
# Automatically compressed → 50-70% fewer tokens
```

---

## License

MIT — See [LICENSE](LICENSE) and [LEGAL_NOTICE.md](LEGAL_NOTICE.md)