Metadata-Version: 2.4
Name: packlm
Version: 1.0.0
Summary: Token-efficient structured data format for LLMs
License: MIT
Project-URL: Homepage, https://github.com/yourusername/packlm
Project-URL: Repository, https://github.com/yourusername/packlm
Project-URL: Issues, https://github.com/yourusername/packlm/issues
Keywords: llm,tokenization,data-format,ai,nlp,anthropic,openai
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# PackLM · ![License: MIT](https://img.shields.io/badge/License-MIT-green.svg) ![Python 3.8+](https://img.shields.io/badge/python-3.8%2B-blue)

> **Token-efficient structured data format for LLMs.**  
> Define fields once, list values — **60–80% fewer tokens than JSON.**

---

## The problem with JSON

Every time you send a list of objects to an LLM, JSON repeats every key name on every row:

```json
[
  {"student_name": "Alice Smith", "age": 20, "grade": "A",  "city": "Delhi"},
  {"student_name": "Bob Kumar",   "age": 19, "grade": "B",  "city": "Mumbai"},
  {"student_name": "Charlie Roy", "age": 21, "grade": "A+", "city": "Delhi"}
]
```
**~60 tokens** — most of it structural noise (`"student_name":` repeated 3× = 12 tokens wasted on one key alone).

## The PackLM way

```
@R student_name age grade city

R "Alice Smith"  20  A   Delhi
R "Bob Kumar"    19  B   Mumbai
R "Charlie Roy"  21  A+  Delhi
```
**~22 tokens** — schema declared once, values only per row.

| | JSON | PackLM | Saving |
|---|---|---|---|
| 3 rows, 4 fields | ~60 tokens | ~22 tokens | **~63%** |
| 100 rows, 4 fields | ~1,800 tokens | ~505 tokens | **~72%** |
| 500 rows, 4 fields | ~9,000 tokens | ~2,505 tokens | **~72%** |

Saving grows **linearly with row count** — the schema cost is paid once.

---

## Installation

```bash
pip install packlm
```

Or just copy `packlm.py` into your project — zero dependencies, standard library only.

---

## Quick start

```python
from packlm import PackLM

students = [
    {"name": "Alice",   "age": 20, "grade": "A",  "city": "Delhi"},
    {"name": "Bob",     "age": 19, "grade": "B",  "city": "Mumbai"},
    {"name": "Charlie", "age": 21, "grade": "A+", "city": "Dehradun"},
]

# Encode
packed = PackLM.encode(students)
print(packed)
# @R name age grade city
# R Alice 20 A Delhi
# R Bob 19 B Mumbai
# R Charlie 21 A+ Dehradun

# Decode back to Python objects
restored = PackLM.decode(packed)
# [{"name": "Alice", "age": "20", "grade": "A", "city": "Delhi"}, ...]
```

---

## Using with LLMs

Add one line to your system prompt and send PackLM instead of JSON:

```python
import anthropic
from packlm import PackLM, SYSTEM_PROMPT

client = anthropic.Anthropic()

packed = PackLM.encode(my_data)

response = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1024,
    system=SYSTEM_PROMPT,   # tells the LLM how to read PackLM
    messages=[{"role": "user", "content": f"Analyse this data:\n\n{packed}"}]
)
```

**System prompt to add to any LLM call:**
```
You receive data in PackLM format.
Lines starting with @ define the schema: @ALIAS field1 field2 ...
Other lines are data rows: ALIAS val1 val2 ... — values match schema order.
~ means null. Multi-word values are double-quoted.
```

---

## Format at a glance

| Syntax | Description | Example |
|---|---|---|
| `@ALIAS f1 f2 f3` | Schema definition — declare once | `@ST name age grade` |
| `ALIAS v1 v2 v3` | Data row — values in schema order | `ST Alice 20 A` |
| `~` | Null / missing value | `ST Bob ~ B` |
| `""` | Explicitly empty string | `ST Charlie "" A` |
| `"value with spaces"` | Multi-word string | `ST "Alice Smith" 20 A` |
| `# comment` | Line comment, ignored by parser | `# student records` |
| *(blank line)* | Ignored, use for readability | |

---

## CLI usage

```bash
# Encode a JSON file to PackLM
python packlm.py encode data.json

# Decode a PackLM file back to JSON
python packlm.py decode data.packlm
```

---

## Token savings estimate

```python
import json
from packlm import PackLM

original = json.dumps(my_data)
packed   = PackLM.encode(my_data)

stats = PackLM.token_savings_estimate(original, packed)
print(stats)
# {"json_tokens": 860, "packlm_tokens": 210, "tokens_saved": 650, "percent_saved": 75.6}
```

---

## Why this matters at scale

At $15 / 1M input tokens (e.g. Claude Opus):

| Monthly data volume | JSON cost | PackLM cost | Saving |
|---|---|---|---|
| 10M tokens | $150 | ~$37 | **$113 / month** |
| 100M tokens | $1,500 | ~$375 | **$1,125 / month** |

---

## Specification

See [SPEC.md](SPEC.md) for the full grammar, edge cases, and rules for implementing PackLM in other languages.

---

## Contributing

PRs welcome! See [issues](https://github.com/yourusername/packlm/issues) for ideas.

Things that would be great:
- JavaScript / TypeScript implementation
- Rust implementation
- Benchmarks against real tokenizers (tiktoken, sentencepiece)
- Streaming decoder

---

## License

MIT — see [LICENSE](LICENSE).
