Metadata-Version: 2.4
Name: toony
Version: 0.1.0rc3
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Typing :: Typed
Classifier: Development Status :: 4 - Beta
Requires-Dist: starlette>=0.47 ; extra == 'asgi'
Requires-Dist: uvicorn>=0.35 ; extra == 'asgi'
Requires-Dist: pydantic>=2.13.5,<3 ; extra == 'pydantic'
Provides-Extra: asgi
Provides-Extra: pydantic
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Summary: Precise TOON 4.4 serialization for Python, with an independent Rust engine
License-Expression: MIT
Requires-Python: >=3.11, <3.16
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/aeespinosao/toony
Project-URL: Issues, https://github.com/aeespinosao/toony/issues
Project-URL: Source, https://github.com/aeespinosao/toony

# Toony

**TOON serialization for Python, powered by an independent Rust engine.**

Toony turns Python data into [TOON](https://toonformat.dev/) text and back, with a
`json`-style API and optional Pydantic integration. Use it to place structured
records in an LLM prompt without changing your models, tool schemas or JSON
protocols.

**Status:** release candidate `0.1.0rc3`, implementing TOON **4.4.0**.
This is an independent project, not the official TOON Python package.

## Install

```sh
python -m pip install toony==0.1.0rc3
uv pip install toony==0.1.0rc3
poetry add toony==0.1.0rc3
```

For Pydantic integration:

```sh
python -m pip install 'toony[pydantic]==0.1.0rc3'
```

Requires **CPython 3.11–3.15**. A matching wheel does not require Rust. Where a wheel
is unavailable, installation builds from source and requires **Rust 1.88+**.
There are no mandatory Python runtime dependencies.

To install directly from source:

```sh
python -m pip install 'git+https://github.com/aeespinosao/toony.git@v0.1.0rc3'
```

## A familiar API

```python
import toony

value = {"users": [{"id": 1, "name": "Ana"}, {"id": 2, "name": "Bo"}]}
text = toony.dumps(value)
print(text)
# users[2]{id,name}:
#   1,Ana
#   2,Bo

assert toony.loads(text) == value
```

`dump` and `load` accept caller-owned streams. `encode` and `decode` are aliases.
Defaults are two-space indentation, comma delimiter, strict decoding and no trailing
newline. `delimiter="\t"` and `delimiter="|"` are also supported.

```python
with open("users.toon", "w", encoding="utf-8") as file:
    toony.dump(value, file)

with open("users.toon", encoding="utf-8") as file:
    restored = toony.load(file)
```

## Use existing Pydantic models

```python
from pydantic import BaseModel
from toony.pydantic import ToonTypeAdapter

class User(BaseModel):
    id: int
    name: str

adapter = ToonTypeAdapter(list[User])
text = adapter.dump_toon([User(id=1, name="Ana")])
users = adapter.validate_toon(text, toon_strict=True, strict=True)
assert users == [User(id=1, name="Ana")]
```

The adapter uses public `TypeAdapter.dump_json()` and `validate_json()` APIs.
Existing serializers, aliases, context and compatible inclusion/exclusion options
continue through JSON mode. No custom model base class or monkey-patching is needed.
TOON grammar strictness (`toon_strict`) and Pydantic strictness (`strict`) are separate.

TOON normalizes values: `1.0` becomes `1` and negative zero becomes `0`. This can
change numeric union selection or strict validation. Pydantic's Decimal-as-string
serialization remains a string. The adapter does not override a validator's own
numeric precision rules.

## LLM context, without changing protocols

```python
import toony

context = toony.dumps({"users": [{"id": 1, "name": "Ana"}]})
prompt = (
    "Read TOON data: array headers declare counts and fields; rows follow field order.\n"
    + context
    + "\nReturn the user IDs as JSON."
)
# Pass prompt to your existing model client.
```

Keep tool arguments, schemas and protocol envelopes in their native JSON contracts.
Send TOON as text when it helps. Avoid passing the same context twice as both TOON
and a second JSON rendering.

Runnable recipes are provided for [LangChain](https://github.com/aeespinosao/toony/blob/v0.1.0rc3/examples/langchain_context.py),
[Pydantic AI](https://github.com/aeespinosao/toony/blob/v0.1.0rc3/examples/pydantic_ai_context.py), [Strands](https://github.com/aeespinosao/toony/blob/v0.1.0rc3/examples/strands_context.py)
and [MCP](https://github.com/aeespinosao/toony/blob/v0.1.0rc3/examples/mcp_context.py). These frameworks are optional application
choices, not dependencies of the core package.

For independent chunks with a verified token budget (this example needs
`pip install tiktoken`; any token-counting function can be supplied):

```python
import tiktoken
from toony.context import iter_chunks

encoding = tiktoken.get_encoding("o200k_base")
records = [{"id": i, "name": f"User {i}"} for i in range(1000)]
for chunk in iter_chunks(
    records, max_tokens=2000, count_tokens=lambda text: len(encoding.encode(text))
):
    assert len(encoding.encode(chunk)) <= 2000
```

Each chunk is a complete document with a valid header/count and preserved IDs.
A single record that exceeds the budget raises `OversizedRecordError`; it is not
silently truncated. Account for your surrounding instructions separately.

## Precision, limits and errors

```python
from decimal import Decimal
import toony

number = Decimal("0.123456789012345678901234567890")
assert toony.loads(toony.dumps(number), parse_float=Decimal) == number
assert toony.loads(toony.dumps(2**100 + 1)) == 2**100 + 1

limits = toony.Limits(max_input_bytes=1_048_576, max_depth=32)
restored = toony.loads("id: 1", limits=limits)
```

Supported inputs include JSON types, tuples (normalized to lists), finite Decimal
values and custom types normalized with `default=`. Dictionary keys must be strings.
Default float decoding follows Python floating-point behavior; use numeric hooks
when exact decimals are required. Non-finite host floats normalize to null;
non-finite Decimal values are rejected.

Default protections are 64 MiB input, 256 MiB output, depth 128, 1,000,000 value
nodes, 16,384 fields per object and 4,096 characters per number. Python's own integer
conversion limits still apply. User callbacks can allocate independently.
`ToonDecodeError`, `ToonEncodeError`, `ToonLimitError`, `ToonNumberError` and
`ToonCycleError` distinguish failure types. Syntax errors include a stable code,
one-based line and bounded excerpt.

Experimental `from_json` and `to_json` convert directly in Rust without creating a
Python value graph. `to_json` returns bytes; JSON duplicate keys are rejected.
These paths may materialize a native tree and are not streaming APIs.

## ASGI / Uvicorn

Install the `asgi` extra and see the [Starlette application](https://github.com/aeespinosao/toony/blob/v0.1.0rc3/examples/asgi_app.py).
`toony.asgi` provides content negotiation, bounded request bodies and helpers that
offload synchronous conversions from the event loop. Uvicorn serves the application:

```sh
python -m pip install 'toony[asgi]==0.1.0rc3'
# Run from a checkout of this repository:
uvicorn examples.asgi_app:app
```

Traversal of Python objects holds the GIL. A thread offload is not a guarantee of
uninterrupted event-loop responsiveness for large CPU-heavy conversions; measure
your workload and use process workers when appropriate. An already-started thread
operation may continue after its awaiting task is cancelled.

## Measured performance

<!-- METRICS:START -->
Measurements below use a local release build of the **0.1.0rc3 sources**, on an **Apple M4
(10 CPU cores, 16 GiB), macOS ARM64, CPython 3.14.7**. Ten independent processes
per codec/operation, seven samples per process, randomized order; intervals are
95% bootstrap intervals over process medians. Codec and inference runs are separate.
The measured native-library hash is recorded in the data. Published wheels are
built in CI; these timings are not measurements of every platform or interpreter.

![Codec latency: Toony compared with JSON and TOON implementations](https://raw.githubusercontent.com/aeespinosao/toony/v0.1.0rc3/assets/benchmarks/codec-latency.png)

For 20,000 uniform records, Toony encoding is **6.36 ms**
and decoding **7.31 ms**. `toons` encodes in
**2.89 ms**; Toony does **not** demonstrate general speed
superiority. JSON codecs produce a different representation. Text outputs are
aligned; orjson's bytes-to-text conversion is included in its encoding time.

Native TOON → JSON text conversion is **8.02 ms**
for Toony versus **10.66 ms** for `toons.to_json`,
a **24.8%** reduction in this workload. This is a
specific measured path, not an across-the-board claim. Total process RSS did not
demonstrate a material improvement.

![Full-prompt token changes including format instructions and negative savings](https://raw.githubusercontent.com/aeespinosao/toony/v0.1.0rc3/assets/benchmarks/prompt-tokens.png)

The token chart counts **the complete prompt**, including the TOON explanation,
against compact JSON. Small and irregular inputs show negative savings.
`o200k_base` is a representation comparison, not either local model's tokenizer.
Exact counts, instructions, both tokenizers (`o200k_base`, `cl100k_base`), dependency
versions and raw timing samples are in [metrics.json](https://github.com/aeespinosao/toony/blob/v0.1.0rc3/assets/benchmarks/metrics.json).
The serialized bytes were verified unchanged from the token-counting run.

**Validation:** 610 tests passed on the locally installed rc3 wheel with CPython
3.14.7. The release workflow tests installed wheels for Python 3.11–3.15 on Linux
x86_64 and macOS ARM64 before publication. Windows remains unvalidated. Earlier
Linux ARM64 tests were executed on rc2. The Rust runtime sources are unchanged;
the rc2 ASAN smoke covered 227,963 executions in 181 seconds. Release CI also runs
a fresh sanitizer smoke. A four-hour campaign on this candidate remains pending.

**Local-model follow-up (executed on 0.1.0rc2):** 336/336 tasks completed; allocated context
16,384, actual comprehension prompts 1012–15272 tokens,
temperature zero, two repetitions and at most one syntax/schema repair.

- `qwen3:8b` comprehension: JSON 36/48; TOON 26/48 correct.
- `llama3.2:3b` comprehension: JSON 28/48; TOON 12/48 correct.

- `qwen3:8b` generation: JSON text 24/24; TOON text 24/24; native JSON schema 24/24 correct.
- `llama3.2:3b` generation: JSON text 12/24; TOON text 0/24; native JSON schema 20/24 correct.

- `qwen3:8b`: 25.1% fewer input tokens; 3.7% more total tokens per correct comprehension task, including all attempts.
- `llama3.2:3b`: 34.2% fewer input tokens; 53.6% more total tokens per correct comprehension task, including all attempts.

These model results characterize the TOON representation; identical serialized
text from another implementation produces the same prompt.

Live framework smokes produced the expected result in **3/8** calls.
Other calls returned incorrect/empty data or no usable structured result. API
contract tests passing does not guarantee model comprehension.

![Local inference latency, separating first and reused prefixes](https://raw.githubusercontent.com/aeespinosao/toony/v0.1.0rc3/assets/benchmarks/inference-latency.png)

Whiskers are 95% bootstrap intervals clustered by source document, not questions
or repetitions. First-prefix requests unload the isolated runner before each document, format and
repetition; they include model loading. The next three questions share that document
prefix. OS file caches are not flushed. Observed cached-token counters are retained
in the data; these phases are not isolated cold-hardware versus warm-hardware tests.

The rc3 runtime code is unchanged apart from version metadata. Serialized
inputs and parsing outcomes were checked against rc2; this model study was not
rerun or relabeled as an rc3 execution.

There are only six independent source documents per model. The non-inferiority
result at a two-percentage-point margin is **inconclusive**. Generation uses an
explicit TOON syntax primer and is a distinct experiment. No energy or monetary
savings were measured. Only one model was resident during scored inference; these are local-host
measurements rather than isolated hardware or energy measurements.
<!-- METRICS:END -->

Token reduction comes from the TOON representation, not the Rust implementation.
Uniform tables can benefit; small or irregular documents can cost more tokens,
especially after format instructions. A shorter prompt does not guarantee better
model accuracy or lower end-to-end latency. Benchmark your data and model before
switching a production workflow.

## Build, test and benchmark

```sh
uv sync --frozen
uv run pytest -q
cargo test -p toony-core --locked
cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
uv run mypy --follow-imports=silent tests/typing_check.py
uv run ruff check python tests examples benchmarks
uv build --wheel --sdist
```

The test suite includes all **564 pinned official fixtures**, property tests,
precision, limits, cycles, callbacks, Pydantic, chunking and HTTP integration.
Run the benchmarks independently of model inference:

```sh
uv run python -m benchmarks.run --repeats 10
uv run --with matplotlib python -m benchmarks.plot
```

The benchmark harness currently measures RSS on macOS/Linux. It uses fresh
processes, seven samples per process, seeded task order, bootstrap intervals and
explicit compatibility exclusions. Raw JSON is retained; no performance threshold
is used as a substitute for correctness.

The release workflow builds and tests installed wheels on Linux and macOS for
Python 3.11–3.15, publishes to TestPyPI, verifies installation, then publishes to
PyPI for matching `v*` tags. Credentials live in GitHub environment secrets.
Windows checks remain available through the manual test workflow; a configured
job does not mean that platform has been validated.

## License and provenance

MIT. The Rust TOON engine is original to this project, written from the specification.
Official compatibility fixtures are pinned in [fixtures/manifest.json](https://github.com/aeespinosao/toony/blob/v0.1.0rc3/fixtures/manifest.json)
and retain their [upstream license](https://github.com/aeespinosao/toony/blob/v0.1.0rc3/fixtures/LICENSE). Runtime/build dependency
licenses are included in [THIRD_PARTY_NOTICES.md](https://github.com/aeespinosao/toony/blob/v0.1.0rc3/THIRD_PARTY_NOTICES.md).

