Metadata-Version: 2.4
Name: tml-renderers
Version: 0.1.0
Classifier: Typing :: Typed
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: License :: OSI Approved :: Apache Software License
License-File: LICENSE
License-File: THIRD_PARTY_LICENSES.md
Summary: Public TML chat types, native tokenizer, and TMLv0 render/parse
Keywords: chat,message,tml
Author-email: Thinking Machines Lab <tinker@thinkingmachines.ai>
License-Expression: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# tml-renderers

`tml-renderers` is the default TML renderer for text and multimodal chat. It packages
public message types, a native tokenizer, TMLv0 rendering and parsing, and media
codecs for inference and training.

## Installation

```bash
pip install tml-renderers
```

`tml-renderers` is a compiled extension that requires PyTorch 2.10 or newer.

## Quickstart

```python
from tml_renderers.chat import Message, MessageList, Text, Author, AuthorKind

message = Message(
    content=Text("Hello, world!"),
    author=Author(AuthorKind.User),
)
messages = MessageList([message])
```

## OpenAI message compatibility

Partners can use OpenAI-shaped message dictionaries at the package boundary
without rewriting their chat data into native classes:

```python
from tml_renderers import chat, tokenizers, v0

openai_message_dicts = [
    {"role": "system", "content": "Answer concisely."},
    {"role": "user", "content": "What is 2 + 2?"},
]

openai_messages = chat.OpenAIMessage.from_oss_messages(openai_message_dicts)
round_tripped = chat.OpenAIMessage.to_oss_messages(openai_messages)

renderer = v0.Renderer(tokenizers.o200k_base_chat())
spans, parser = renderer.render_for_completion(openai_messages)
```

The compatibility surface covers role/content messages, multipart text and
`image_url` content, assistant tool calls, and tool results.
`render_for_completion` and `render_for_sft` also accept `list[OpenAIMessage]`
directly. This is a message-format bridge; it does not implement the OpenAI
network client or server API. Use `MessageList.from_oss_messages(...)` and
`MessageList.to_oss_messages()` when you want the same conversion through a
native `MessageList`.

## Rendering and parsing (TMLv0)

Rendering converts structured chat messages into the exact token and media-span
sequence consumed by the model. `v0` uses the same TMLv0 grammar for
both inference and training:

- `render_for_completion(...)` renders an inference prompt and returns a parser
  for turning generated model tokens back into messages.
- `render_for_sft(...)` renders supervised-training examples with aligned,
  per-token loss weights.

The parser performs the reverse operation, reconstructing structured messages
from generated token spans:

```python
from tml_renderers import chat, tokenizers, v0

tokenizer = tokenizers.o200k_base_chat()
renderer = v0.Renderer(tokenizer)

messages = [
    chat.Message(
        content=chat.Text("Hello"),
        author=chat.Author(chat.AuthorKind.User),
    )
]

spans, parser = renderer.render_for_completion(messages)
parsed = parser.parse(spans)

training_examples = renderer.render_for_sft(messages)
```

The tokenizer is a native `o200k` BPE tokenizer plus the fixed TMLv0 special
tokens. No Hugging Face chat template or remote code is involved.

```python
tokenizer.encode_ordinary("hello")
tokenizer.decode([24912])
tokenizer.encode_special("message_user")   # -> 200000
tokenizer.decode_special(200000)            # -> "message_user"
```

Full method surface:

| method | maps | on invalid / partial UTF-8 |
|--------|------|----------------------------|
| `encode_ordinary(text)` | text → ids | — (special-token markup is encoded as literal text) |
| `encode_special(name)` | name → id | raises if `name` isn't a special token |
| `decode(ids)` | ids → text | **lossy** — invalid/partial bytes become `U+FFFD`, never raises (tiktoken parity) |
| `decode_strict(ids)` | ids → text | **raises** `ValueError` on invalid/partial UTF-8 |
| `decode_bytes(ids)` | ids → bytes | raw bytes, no UTF-8 handling |
| `decode_special(id)` | id → name | `None` if `id` isn't a special token |

Plus `is_special_token(id)`, `special_tokens()`, and the `eos_token` / `bos_token` /
`all_special_tokens` properties. `decode` is **lossy** so a truncated multi-byte tail (common
when streaming) renders as `�` rather than raising — reach for `decode_strict` only when you
want that raise.

`render_for_completion` accepts a `MessageList`, a `list[Message]`, or a
`list[OpenAIMessage]`. `render_for_completion_with_effort(messages, effort)` (with
`effort` a float in `[0, 1]`) inserts a system message containing
`Thinking effort level: 0.9` before the first non-system message; the messages
must not already contain a `ThinkingEffort` content. For SFT,
`render_for_sft` returns `TrainingExample`s carrying per-token loss weights.

### Media

`tml-renderers` includes the multimodal codec logic used by the default TMLv0
renderer:

- An `ImagePointer` is mapped to the model's black-padded image patch layout and
  rendered as an `ImageAssetPointerTokenSpan` with the expected image-token
  count.
- An `AudioPointer` is decoded, resampled to 16 kHz, DMel-encoded, and rendered
  as a `DmelTokenSpan`.

Image and audio messages use the same completion and training APIs as text:

```python
from tml_renderers import chat, tokenizers, v0

renderer = v0.Renderer(tokenizers.o200k_base_chat())

image_msg = chat.Message(
    content=chat.ImagePointer(
        location="image.png",
        format=chat.ImageFormat.Png,
        width=512,
        height=512,
    ),
    author=chat.Author(chat.AuthorKind.User),
)

audio_msg = chat.Message(
    content=chat.AudioPointer(
        location="clip.wav",
        format=chat.AudioFormat.Wav,
        num_frames=48_000,
        sample_rate=16_000,
    ),
    author=chat.Author(chat.AuthorKind.User),
)

spans, _ = renderer.render_for_completion([image_msg, audio_msg])
dmel = next(s.span for s in spans if type(s.span).__name__ == "DmelTokenSpan")
print(dmel.dmel.shape)  # [num_audio_tokens, num_dmel_bins], uint8
```

Media locations are represented as pointers. Audio is read from local files;
`tml-renderers` never fetches remote media.

### Streaming

`Parser.parse_token` / `parse_updates` / `flush_updates` return incremental
`ParseUpdate`s, so you can render a completion as tokens arrive rather than only
returning completed `Message`s. Each update is a `StreamingMessageHeader`, a
`StreamingContent` delta, or a completed `Message`. See
`tests/test_v0_streaming.py` for a worked example.

## Token spans

Renderers work with token-span types you can construct directly:

```python
from tml_renderers.chat import (
    EncodedTextTokenSpan,
    ImageAssetPointerTokenSpan,
    ImageFormat,
    PaddingTokenSpan,
    TokenSpan,
)

text = EncodedTextTokenSpan([101, 102])
image = ImageAssetPointerTokenSpan(
    location="image.png", format=ImageFormat.Png, width=512, height=512, num_tokens=256
)
padding = PaddingTokenSpan(4)

# Wrap in the oneof only where a call boundary asks for a TokenSpan.
wrapped = TokenSpan(text)
```

`DmelTokenSpan` is the encoded-audio span produced by the renderer's audio
handler; `dmel` is a `uint8` tensor of shape `[num_audio_tokens, num_dmel_bins]`.

## Using with Tinker

Install the latest Tinker SDK and Tinker Cookbook, then select the
`tml_v0` renderer in your Cookbook recipe. Cookbook handles model-specific
configuration and converts rendered text, image, and audio inputs for sampling
and SFT.

The lower-level `tml_renderers.tinker` helpers are also available when integrating
directly with the Tinker SDK. Remote media is never fetched; provide local files
or base64 image data.

## Types

- **Core**: `Message`, `MessageList`, `Author`, `AuthorKind`, `MessageChannel`,
  `MessageMetadata`
- **Content**: `Text`, `Thinking`, `ThinkingEffort`, `ImagePointer`,
  `AudioPointer`, `InvokeTool`, `StructuredToolCall`, `ToolError`,
  `ToolDeclareJson`, `ModelEndSampling`
- **Token spans**: `TokenSpan`, `EncodedTextTokenSpan`,
  `ImageAssetPointerTokenSpan`, `DmelTokenSpan`, `PaddingTokenSpan`
- **Formats**: `ImageFormat`, `AudioFormat`
- **Tool args**: `ToolArg`, `ToolMetadata`. `ToolArg.value` is a JSON-encoded
  value string, matching the OpenAI `function.arguments` convention.
- **Tool declarations**: `ToolSpecJson` (one declared tool: `name` /
  `description` / `type_name`, plus `parameters` as a JSON-schema string; the
  constructor accepts `parameters` as a dict and serializes it), grouped into a
  `ToolDeclareJson`.

## Examples

- `examples/run_tml_v0.py` — low-level render/parse of text, tools, and audio.
- `examples/cookbook_tinker_demo.py` — OpenAI-style dicts alongside native types.

## License

Apache-2.0. See [`LICENSE`](LICENSE).

