Metadata-Version: 2.1
Name: p-toon-llm
Version: 0.1.1
Summary: Token-oriented object notation (TOON) for compact, reversible JSON encoding
Author: Pharen IT GmbH
License: MIT License
        
        Copyright (c) 2026 Pharen IT GmbH
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://example.com
Project-URL: Repository, https://example.com
Keywords: toon,json,serialization,llm,tokens
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"

# TOON Format (Python)

TOON is a compact, reversible encoding of JSON designed to reduce LLM prompt tokens.
This implementation focuses on a minimal, deterministic syntax and fast encode/decode.

## Core Syntax

1. Objects use a key list and a value list: `{k1,k2|v1|v2}`
2. Arrays use pipe separators: `[v1|v2|v3]`
3. Tables (uniform dict arrays) use CSV blocks: `^csv[k1,k2|r1c1,r1c2|r2c1,r2c2]`

TOON is best for mixed or nested data. Pure tabular data is still smaller in raw CSV.

## Quick Start

```python
from toon_format import encode, decode, encode_best, encode_as, convert_format

data = {"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}

print(encode(data))
print(decode(encode(data)))
print(encode_best(data))
print(encode_as(data, "json"))
print(convert_format(encode(data), "yaml"))
```

## Installation

1. Install the package from PyPI:

```
pip install p-toon-llm
```

2. Optional dev tools for build and upload:

```
pip install -e .[dev]
```

## Auto Mode

1. Pick the smallest format automatically (tokens by default):

```python
best = encode_best(data, candidates=("toon", "json", "yaml", "csv"))
print(best["format"], best["text"])
```

2. Or use auto mode inside `encode()`:

```python
print(encode(data, options={"mode": "auto", "candidates": ("toon", "json", "csv")}))
```

## Benchmarks

This repository provides tooling to measure token savings locally so you can benchmark your own data.
Run the demo or use `compare_formats` for a quick comparison, then expand to your real datasets.

Example workflow:

1. Start with a representative dataset that matches your production structure.
2. Compare TOON vs JSON compact, JSON pretty, YAML, and CSV.
3. Measure both tokens and chars, and pick the best format for your use case.

```python
from toon_format import compare_formats, estimate_savings

print(compare_formats(data))
print(estimate_savings(data))
```

If you want to publish benchmark results in this README, add your measured numbers and dataset description.

## Format Examples

Example object:

```
{name,age|Alice|30}
```

Example array:

```
[1|2|3]
```

Example table:

```
^csv[id,name|1,Alice|2,Bob]
```

## Layout

```
src/
  encoder.py
  decoder.py
  detect.py
  tokens.py
  compare.py
  formats.py
  convert.py
  toon_format.py
```

## Docs

1. `docs/index.md`
2. `docs/format.md`
3. `docs/api.md`
4. `docs/performance.md`
5. `docs/prompt_header.md`

## Tests

```
python -m pytest
```

## Demo

```
python examples/demo.py
```
