Metadata-Version: 2.5
Name: llm-dialects
Version: 0.1.0
Summary: Streaming parsers for the tool-call and reasoning dialects of open-weight LLMs. Zero dependencies.
Project-URL: Homepage, https://github.com/cb-Sigma/llm-dialects
Project-URL: Documentation, https://github.com/cb-Sigma/llm-dialects/tree/main/docs
Project-URL: Issues, https://github.com/cb-Sigma/llm-dialects/issues
Project-URL: Changelog, https://github.com/cb-Sigma/llm-dialects/blob/main/CHANGELOG.md
Author: The llm-dialects Authors
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: deepseek,function-calling,llama,llm,mistral,openai-compatible,parser,qwen,reasoning,streaming,tool-calling
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# llm-dialects

**Streaming parsers for the tool-call and reasoning formats of open-weight LLMs. Raw text in, structured deltas out. Zero dependencies.**

[![CI](https://github.com/cb-Sigma/llm-dialects/actions/workflows/ci.yml/badge.svg)](https://github.com/cb-Sigma/llm-dialects/actions/workflows/ci.yml)
[![License](https://img.shields.io/github/license/cb-Sigma/llm-dialects)](LICENSE)
[![Python](https://img.shields.io/python/required-version-toml?tomlFilePath=https%3A%2F%2Fraw.githubusercontent.com%2Fcb-Sigma%2Fllm-dialects%2Fmain%2Fpyproject.toml)](pyproject.toml)

```python
from llm_dialects import parser

p = parser("qwen3", tools=tools)          # or parser.detect(chat_template=..., model_id=...)
for delta in text_deltas:                 # raw model text, in chunks of any size
    for ev in p.feed(delta): ...          # ReasoningDelta, ContentDelta, ToolCallStart, ToolCallArgsDelta, ToolCallEnd
for ev in p.close(): ...
```

Fed with what Qwen3 actually generates, nine characters at a time —

```text
<think>\nThe user wants the weather in Paris.\n</think>\n\nLet me check.\n<tool_call>\n{"name": "get_weather", "arguments": {"city": "Paris"}}\n</tool_call>
```

— the loop yields:

```python
ReasoningDelta(text="T")
ReasoningDelta(text="he user w")
ReasoningDelta(text="ants the")
ReasoningDelta(text=" weather i")
ReasoningDelta(text="n Paris.")
ContentDelta(text="Let me c")
ContentDelta(text="heck.")
ToolCallStart(index=0, id="call_0", name="get_weather", known=True)
ToolCallArgsDelta(index=0, text="{")
ToolCallArgsDelta(index=0, text='"city": "Paris"}')
ToolCallEnd(index=0, arguments='{"city": "Paris"}', status="ok")
```

No markup leaks, no half-emitted `<tool_c`, and the argument fragments concatenate to valid JSON. The same events come out for any chunking of the same text.

```bash
pip install git+https://github.com/cb-Sigma/llm-dialects
```

Python 3.10+. Not on PyPI yet.

## Contents

- [Why this exists](#why-this-exists)
- [Supported dialects](#supported-dialects)
- [The chunk-boundary guarantee](#the-chunk-boundary-guarantee)
- [When the model misbehaves](#when-the-model-misbehaves)
- [API in one screen](#api-in-one-screen)
- [Building an OpenAI-compatible server](#building-an-openai-compatible-server)
- [The conformance corpus](#the-conformance-corpus)
- [Performance](#performance)
- [Limitations and maintenance reality](#limitations-and-maintenance-reality)
- [Related work](#related-work)
- [FAQ](#faq)
- [Roadmap](#roadmap)
- [Contributing](#contributing)

## Why this exists

Open-weight models do not emit tool calls or reasoning as structured data. They emit text, in a format ("dialect") that each model family invented for itself:

```text
<tool_call>{"name": "f", "arguments": {...}}</tool_call>             Qwen, Hermes
<|python_tag|>{"name": "f", "parameters": {...}}                     Llama 3.x
[f(a=1, b="x")]                                                      Llama 4
[TOOL_CALLS]f[ARGS]{...}                                             Mistral
<start_function_call>call:f{a:<escape>x<escape>}<end_function_call>  FunctionGemma
<tool_call>f<arg_key>a</arg_key><arg_value>x</arg_value></tool_call> GLM
<|channel|>commentary to=functions.f<|message|>{...}<|call|>         gpt-oss
```

Turning that text into `reasoning_content` / `content` / `tool_calls` deltas *while it streams* is re-implemented inside every inference engine — vLLM, SGLang, llama.cpp, Ollama, mlx-lm — and it is a steady source of bugs. On 2026-11-13 the vLLM issue tracker listed 389 issues labelled `tool-calling` and 60 with "streaming tool parser" in the title; the recent ones are the same few failures in one parser after another: streaming and non-streaming paths disagreeing, text leaking into or out of arguments, content dropped after a marker, a call lost because a closing tag was missing. A few that were read in full for this project:

- [vllm#39056](https://github.com/vllm-project/vllm/issues/39056): a tool call emitted before `</think>` ends up in `reasoning`, and `tool_calls` stays empty.
- [mlx-lm#1627](https://github.com/ml-explore/mlx-lm/issues/1627): the Qwen3-Coder parser ran `int("140.0")`, raised, and the server silently dropped the call while still reporting `finish_reason: "tool_calls"`; an agent looped ~390 times.
- [mlx-lm#1096](https://github.com/ml-explore/mlx-lm/issues/1096), [mlx-lm#1885](https://github.com/ml-explore/mlx-lm/issues/1885): a new model family (Gemma 4, MiniCPM5) simply had no parser, so calls arrived as plain text.
- [llama-cpp-python#1784](https://github.com/abetlen/llama-cpp-python/issues/1784) (open since 2024) and [#2227](https://github.com/abetlen/llama-cpp-python/issues/2227): Qwen2.5 and Gemma 4 tool calls come back as raw text in `content`, because the Python binding has no output-side parser.

If you run one of the big engines' own servers, you already have their parsers. But if you write your own OpenAI-compatible endpoint — on MLX, transformers, llama-cpp-python, LitServe, a raw `/completions` endpoint — or post-process raw completions, there has been nothing small to import. The engines' parsers are wired into their servers and tokenizers. The one pip-installable extraction, [llm-router-utils](https://github.com/wxsms/llm-router-utils) (SGLang's parsers), depends on pinned `transformers`, `openai`, `fastapi` and on `xgrammar`, which requires `torch`.

llm-dialects is that small thing: a few thousand lines of typed Python with no dependencies, built around two guarantees that are tested rather than hoped for — it never emits text it would have to take back, and its output does not depend on how the stream was chunked.

Two things upstream matter for where this is going, and they are not hidden here: llama.cpp now derives its parsers from the chat template, and Hugging Face transformers is introducing *response templates* that models can ship. See [Limitations and maintenance reality](#limitations-and-maintenance-reality).

## Supported dialects

"Real output" means the corpus contains raw text captured from that model on this project's own runs, and the parser is tested on it (token by token, as generated). "Engine vectors" means the dialect is tested against test vectors adapted from the engines' own test suites and against the layout its chat template renders, but has **not** been run against a live model here.

| Dialect | Model families | Reasoning | Tool calls | Validated on | Sources |
|---|---|:-:|:-:|---|---|
| [`hermes`](docs/dialects/hermes.md) (`qwen`, `qwen3`, `chatml`) | Qwen2.5, Qwen3, Hermes 2 Pro / 3, SmolLM3 | `<think>` | ✓ | **real output**: Qwen2.5-0.5B, Qwen3-0.6B; engine vectors | vLLM, SGLang, llama.cpp, Ollama, templates |
| [`think`](docs/dialects/think.md) | DeepSeek-R1 distills, QwQ, any `<think>` model | `<think>` | – | **real output**: DeepSeek-R1-Distill-Qwen-1.5B; engine vectors | vLLM, SGLang, Ollama |
| [`llama3-json`](docs/dialects/llama3-json.md) | Llama 3.1 / 3.2 / 3.3 | – | ✓ | **real output**: Llama-3.2-1B; engine vectors | vLLM, SGLang, templates |
| [`gemma`](docs/dialects/gemma.md) (`functiongemma`, `gemma4`) | FunctionGemma, Gemma 4 | Gemma 4 thought channel | ✓ | **real output**: FunctionGemma-270M; Gemma 4 on engine vectors only | vLLM, SGLang, Ollama, templates |
| [`deepseek`](docs/dialects/deepseek.md) (`deepseek-r1`, `deepseek-v3.1`) | DeepSeek-V3, R1, V3.1 | `<think>` | ✓ | engine vectors only (tool format) | vLLM, SGLang, templates |
| [`pythonic`](docs/dialects/pythonic.md) (`llama4`) | Llama 4, Llama 3.2 pythonic prompt | – | ✓ | engine vectors only | vLLM, SGLang, template |
| [`mistral`](docs/dialects/mistral.md) (`magistral`, `devstral`) | Mistral 7B v0.3, Nemo, Small 3.x, Magistral | `[THINK]` | ✓ both generations | engine vectors only | vLLM, SGLang, llama.cpp, Ollama, templates |
| [`granite`](docs/dialects/granite.md) | Granite 3.0–3.3, 4.0 | `<think>` + `<response>` (3.3) | ✓ | engine vectors only | vLLM, templates |
| [`phi4`](docs/dialects/phi4.md) | Phi-4-mini | – | ✓ | engine vectors only | vLLM, llama.cpp |
| [`glm`](docs/dialects/glm.md) | GLM-4.5 / 4.6 / 4.7 | `<think>` | ✓ | engine vectors only | vLLM, SGLang, llama.cpp, Ollama, templates |
| [`kimi-k2`](docs/dialects/kimi-k2.md) | Kimi K2 | `<think>` | ✓ | engine vectors only | vLLM, SGLang, template |
| [`minimax-m2`](docs/dialects/minimax-m2.md) | MiniMax-M2 | `<think>`, interleaved | ✓ | engine vectors only | vLLM, SGLang, template |
| [`qwen3-coder`](docs/dialects/qwen3-coder.md) | Qwen3-Coder | `<think>` | ✓ | engine vectors only | vLLM, SGLang, Ollama, template |
| [`harmony`](docs/dialects/harmony.md) (`gpt-oss`) | gpt-oss-20b / 120b | analysis channel | ✓ | engine vectors only; text-level, see note | Ollama, SGLang, template |

Each dialect page has the grammar, the markers, the exact source files it was derived from, and the cases where it deliberately differs from an engine. Sources are pinned to commits in [docs/sources.md](docs/sources.md).

**Harmony:** this is a text-level parser checked against vectors from Ollama's Harmony tests. Harmony is defined on tokens; when you have token ids, the official [openai-harmony](https://pypi.org/project/openai-harmony/) is the reference and the better choice. It is here so that servers with only rendered text can use one API for every model.

Programmatic access to all of this: `llm_dialects.dialects()` returns the registry with models, sources and validation level.

## The chunk-boundary guarantee

For any input and any way of cutting it into chunks:

1. **Nothing is emitted that might be taken back.** A marker split across chunks (`<tool_` + `call>`) is held, minimally, until it is decided; a comma, an object key, a number are held until what follows makes them certain. Everything else passes through at once — a 200 kB string argument streams in pieces as it arrives.
2. **The canonical event sequence is identical** — and `parse(text)` returns exactly that sequence. ("Canonical" = adjacent text deltas merged, `llm_dialects.coalesce`; how text is cut into deltas necessarily follows the chunks.)
3. **`ToolCallArgsDelta` fragments of a call always concatenate to valid JSON**, whatever the model wrote.

This holds by construction — each parser is a generator that reads characters and `yield`s when it needs more, so it cannot see chunk boundaries, and `parse()` is literally `feed()` + `close()` on the same machine ([docs/design.md](docs/design.md)) — and it is tested, because by-construction arguments have gaps (the first property-test run found one):

| Test | What it checks |
|---|---|
| every split point | each of the 222 corpus cases, cut at every single position, gives the same events and the same issues |
| one character at a time | each corpus case fed character by character |
| recorded token chunks | each real case replayed with the exact token pieces the model produced |
| random multi-way splits | seeded sweeps and [hypothesis](https://hypothesis.readthedocs.io)-drawn cut points over the corpus |
| render → parse round trip | hypothesis generates responses (reasoning, content, calls with nested JSON arguments), a test-side *writer* renders them in each dialect, and the parser must return the same structure for every chunking; 16 writers cover 13 of the 14 dialects (`think` is exercised through the reasoning part of the others) |
| truncation anywhere | the rendered responses cut short at a random point: still valid, still chunk-invariant |
| marker soup | random concatenations of every marker of every dialect, JSON fragments and junk, in all three reasoning modes: `feed()` never raises, arguments are valid JSON, events and issues are chunk-invariant |
| mutated corpus | corpus inputs with spans deleted, duplicated or spliced |
| plain text | text without markup comes back as content byte for byte |
| strict policy | under `STRICT`, a case either parses to the same events or raises the same error at the same position for every chunking |
| JSON transcoder | valid JSON passes through verbatim; every prefix of a valid document becomes a valid document; arbitrary text never produces invalid output |

In numbers: the suite has 1,832 tests and passes on Python 3.10–3.13. The corpus sweep parses 40,034 distinct single-cut chunkings of the 222 cases and replays 4,736 recorded model tokens; 95 property tests draw 9,500 generated inputs per run by default (19,000 in CI, and 190,000 with `HYPOTHESIS_PROFILE=thorough`, which was run before this release). `mypy --strict` and `ruff` are clean.

## When the model misbehaves

Small and quantised models produce broken output routinely; large ones do it at `max_tokens`. What happens then is a policy, explicit and configurable:

```python
from llm_dialects import Policy, STRICT, parser

p = parser("qwen3", tools=tools)  # LENIENT (default): never raises
p = parser("qwen3", tools=tools, policy=STRICT)  # raise on every deviation
p = parser(
    "qwen3", tools=tools, policy=Policy(malformed_arguments="raise", text_after_tool_call="drop")
)
```

| Situation | `Policy` field | Lenient (default) | Other choices |
|---|---|---|---|
| Arguments are not valid JSON | `malformed_arguments` | `"repair"`: fix what is unambiguous → `status="repaired"`; otherwise close what was streamed into valid JSON → `status="invalid"` | `"raise"` → `ArgumentsError` |
| Stream ends inside a tool call | `truncated_tool_call` | `"repair"`: close strings and brackets → `status="truncated"` | `"raise"` → `TruncatedError` |
| Call to an undeclared tool | `unknown_tool` | `"flag"`: `ToolCallStart.known=False` | `"raise"` → `UnknownToolError` |
| Text after a tool call | `text_after_tool_call` | `"content"`: emitted as content | `"drop"`, `"raise"` |
| Tool call before reasoning closed | `tool_call_in_reasoning` | `"extract"`: ends the reasoning, parses the call | `"reasoning"` (leave it in the reasoning text), `"raise"` |
| Stream ends inside reasoning | `unclosed_reasoning` | `"accept"` | `"raise"` |

Everything handled leniently is also recorded in `parser.issues` (`code`, `message`, `position`), and `ParseResult.ok` is true only if nothing was flagged.

What "repair where unambiguous" covers — each of these appears in the corpus, several in real model output:

```text
{'city': 'Paris', 'days': None,}              single quotes, Python literals, trailing comma
{city: "Paris"}                               unquoted keys
{"text": "line one⏎line two"}                 raw newline inside a string
{"name": "f", "arguments": {"a": 1}</tool_call>    a closing brace short of valid
{{"name": "f", "arguments": {...}}}           doubled braces (Qwen2.5-0.5B does this with the official GGUF template)
```

Truncation never invents data: `{"a": 1, "b":` + end of stream becomes `{"a": 1}`, not `{"a": 1, "b": null}`.

A parse that is syntactically valid is still not a decision to execute. `known=False` and `status != "ok"` are yours to act on; see [SECURITY.md](SECURITY.md).

## API in one screen

```python
from llm_dialects import parser, parse, detect, dialects, coalesce, Policy, LENIENT, STRICT

p = parser("mistral", tools=tools, policy=LENIENT, reasoning=None, id_factory=None)
p.feed(text) -> list[Event]        # events that became certain
p.close()    -> list[Event]        # end of stream: flush, close open calls
p.issues     -> list[Issue]        # what was repaired or flagged

result = parse(text_or_chunks, "mistral", tools=tools)   # same machine, same result
result.content, result.reasoning, result.tool_calls, result.events, result.issues, result.ok
call = result.tool_calls[0]; call.name, call.id, call.arguments, call.json(), call.status, call.known

found = detect(chat_template=tokenizer.chat_template, model_id="Qwen/Qwen3-0.6B")
found.dialect, found.reasoning, found.confidence, found.sure, found.reasons
p = parser.detect(chat_template=..., model_id=...)       # raises if unknown, warns if a guess
```

- **Sync core, async-friendly.** `feed()` does no I/O and never blocks; call it inside `async for` ([examples/async_stream.py](examples/async_stream.py)). One parser per response.
- **`tools=`** takes the OpenAI-format tool list of the request. It enables unknown-tool flagging and, for the dialects whose arguments are untyped text (GLM, MiniMax, Qwen3-Coder), gives each value its declared JSON type.
- **`reasoning=`** is `"auto"` (reasoning starts at an opening marker), `"prefilled"` (the chat template already opened the block, so output starts in reasoning — DeepSeek-R1, MiniMax-M2, GLM-4.7) or `"off"`. Each dialect defaults to what its reference template does; `detect()` reads it off a template.
- **Ids** are deterministic (`call_0`, `call_1`, …) unless the dialect carries a native id (Kimi K2, Mistral `[CALL_ID]`). Pass `id_factory=llm_dialects.random_call_id` for globally unique ones.
- **`detect()` says when it is unsure.** A chat template with a known signature is `confidence="high"`. A model id alone is at best `"medium"` — fine-tunes keep names and change templates — and `parser.detect()` warns in that case.

## Building an OpenAI-compatible server

The adapters in `llm_dialects.openai` produce plain dicts in the Chat Completions shapes, so a custom server needs a few lines per endpoint ([examples/openai_server_fastapi.py](examples/openai_server_fastapi.py) is the complete file):

```python
from llm_dialects import parser, random_call_id
from llm_dialects.openai import ChunkStream, completion
from llm_dialects.result import collect


@app.post("/v1/chat/completions")
async def chat_completions(body: dict):
    machine = parser("qwen3", tools=body.get("tools"), id_factory=random_call_id)
    deltas = generate(body["messages"], body.get("tools"))  # your engine: raw text deltas

    if not body.get("stream"):
        events = [ev for delta in deltas for ev in machine.feed(delta)] + machine.close()
        return completion(collect(events, machine.issues), model=body["model"])

    async def sse():
        stream = ChunkStream(model=body["model"])
        for delta in deltas:
            for chunk in stream.chunks(machine.feed(delta)):
                yield f"data: {json.dumps(chunk)}\n\n"
        for chunk in stream.chunks(machine.close()):
            yield f"data: {json.dumps(chunk)}\n\n"
        yield f"data: {json.dumps(stream.finish())}\n\n"  # finish_reason: "tool_calls" or "stop"
        yield "data: [DONE]\n\n"

    return StreamingResponse(sse(), media_type="text/event-stream")
```

Reasoning goes to `reasoning_content` (pass `reasoning_field="reasoning"` for the newer spelling). With llama-cpp-python, use raw completion and let the GGUF's own chat template render the prompt: [examples/llama_cpp_python_tools.py](examples/llama_cpp_python_tools.py).

Two things your server still has to do: make the tokenizer render special tokens (`skip_special_tokens=False`, `special=True`) — a parser cannot see markers that were stripped — and pass `finish_reason="length"` to `finish()` when generation hit the token limit.

## The conformance corpus

[`corpus/`](corpus) holds 222 cases as plain JSON, one file per dialect: the raw input, the tools of the request, the expected segments, and where the input came from.

```json
{
  "id": "kimi-k2/truncated",
  "input": "<|tool_calls_section_begin|><|tool_call_begin|>functions.get_weather:0<|tool_call_argument_begin|>{\"city\": \"Bei",
  "expected": [
    {"type": "tool_call", "name": "get_weather", "arguments": {"city": "Bei"}, "id": "functions.get_weather:0", "status": "truncated"}
  ],
  "provenance": {"kind": "engine-test", "source": "vllm:tests/tool_parsers/test_kimi_k2_tool_parser.py test_truncated_tool_call_no_end_marker"},
  "divergences": [{"engine": "vllm", "behaviour": "streamed arguments are '{\"city\": \"Bei' (asserted by the test), which is not JSON", "reason": "arguments are always valid JSON; the call is flagged 'truncated'"}]
}
```

| Provenance | Cases | Meaning |
|---|--:|---|
| `real` | 48 | captured from a real model through its own chat template, with the token pieces as generated |
| `engine-test` | 110 | adapted from the test suites of vLLM, SGLang, llama.cpp and Ollama (attribution in [NOTICE](NOTICE)) |
| `engine-source` | 10 | an example or documented transition from an engine's parser source |
| `upstream-issue` | 2 | the scenario of a reported engine bug |
| `template` | 27 | the layout the model's chat template renders |
| `synthetic` | 25 | written to pin down an edge case |

The format ([docs/corpus-format.md](docs/corpus-format.md)) does not depend on this library, so other implementations can test against it; [examples/check_other_implementation.py](examples/check_other_implementation.py) is a 40-line harness. 24 cases record an engine that behaves differently, and why the corpus expects something else; they are tabulated, together with the places where engines disagree *with each other*, in [docs/engine-differences.md](docs/engine-differences.md).

**To contribute a case:** capture the raw text (raw completion, special tokens visible — [`scripts/capture_gguf.py`](scripts/capture_gguf.py) does it for GGUF models), add an object to `corpus/<dialect>.json`, write `expected` by hand, state the provenance honestly. All chunk-boundary tests pick it up automatically.

## Performance

Parsing is not a bottleneck: a model streams roughly 0.001 MB/s of text per request.

| Workload (2 MB of text) | 4-character chunks (≈ one token) | 64-character chunks | 4096-character chunks |
|---|--:|--:|--:|
| plain content | 2.2 MB/s | 24.8 MB/s | 664 MB/s |
| reasoning + content | 2.3 MB/s | 25.0 MB/s | 820 MB/s |
| many small JSON tool calls (`hermes`) | 1.8 MB/s | 5.5 MB/s | 6.9 MB/s |
| one large JSON tool call (`hermes`) | 2.6 MB/s | 14.2 MB/s | 21.3 MB/s |
| one large XML tool call (`qwen3-coder`) | 1.4 MB/s | 16.1 MB/s | 152 MB/s |
| pythonic calls | 2.2 MB/s | 6.9 MB/s | 7.8 MB/s |
| Mistral v11 calls | 2.0 MB/s | 6.5 MB/s | 10.8 MB/s |

At one token per `feed()` that is 350,000–650,000 tokens per second on one core — three orders of magnitude above what a model produces for a single stream.

Measured with [`benchmarks/throughput.py`](benchmarks/throughput.py) on CPython 3.12, one core of a shared 4-core development VM (so treat the figures as a floor). With token-sized chunks the cost is dominated by the per-call overhead of `feed()`; JSON arguments are validated character class by character class, which is why they are slower than plain text.

## Limitations and maintenance reality

- **New dialects arrive with model releases.** A parser library is never finished; the vLLM tree alone has more than fifty tool-parser modules, many added in the last year. This project covers the formats in wide use that could be verified from source, and will always trail. The contribution path for a new dialect is deliberately strict (sources, corpus cases, a writer for the property tests) so that what is here stays right.
- **Ten of fourteen dialects have not been run against a live model here.** The big-model formats (DeepSeek, Mistral, GLM, Kimi, MiniMax, Qwen3-Coder, gpt-oss, Llama 4, Granite, Phi-4-mini, Gemma 4) are tested against engine test vectors and template layouts only. Real output from small models already changed the `hermes` and `gemma` parsers; expect the same when these meet reality. Corpus cases from real runs are the most valuable contribution.
- **Engines are evolving their own parsers.** llama.cpp replaced most hand-written parsers with an *auto-parser* that derives the format from the chat template by differential analysis ([docs/autoparser.md](https://github.com/ggml-org/llama.cpp/blob/master/docs/autoparser.md), since March 2026). vLLM moved to a declarative parser engine (`vllm/parser/`), has a Rust front-end in progress ([RFC #40846](https://github.com/vllm-project/vllm/issues/40846), experimental) and reads Hugging Face `response_template` metadata. If you serve with those engines, use their parsers; this library is for everything built outside them.
- **Hugging Face response templates may make hand-written dialects unnecessary.** transformers documents `tokenizer.parse_response()` and a streaming `get_response_parser()` driven by a `response_template` that a model ships next to its chat template. Where a model ships one and you already depend on transformers, that is the model author's own description of the format and should win. What remains for this library: environments without transformers (GGUF, MLX, raw completion APIs), models that ship no response template, valid-JSON argument streaming (response templates stream tool-call bodies as raw "dirty" chunks and parse at region close), and explicit robustness policies. Reading a `response_template` as a dialect definition is on the roadmap.
- **Text-level, not token-level.** A model that *writes about* `<tool_call>` in prose is indistinguishable from one that emits the special token. Engines with token ids can tell; this library cannot. `reasoning="off"` and the fallback-to-content rules limit the damage.
- **The parser cannot see the prompt.** Whether a template pre-opened `<think>` is not in the output. Defaults follow the reference templates; `detect()` and `reasoning=` cover the rest; a wrong setting shows up as a `stray_reasoning_close` issue.
- **No schema validation or coercion.** Arguments are valid JSON, not valid against your schema.
- **Not supported yet:** Phi-4-mini's `<|tool_call|>` form (only the `functools[...]` form that vLLM and llama.cpp test could be verified), Cohere Command, InternLM, LFM2, xLAM, Hunyuan, Step, Ernie, Olmo, MiniCPM5, MiniMax-M3, DeepSeek V3.2/V4 (DSML), Kimi K3, Jamba, Granite 3.2 prose-marker reasoning, Granite 20B function calling. See the roadmap.
- **One edge of the no-buffering rule:** if a model writes `"arguments"` before `"name"`, the arguments are held until the name arrives.

This is an independent project. It is not affiliated with, sponsored by or endorsed by any inference engine or model vendor; model and product names are used only to say which format is which.

## Related work

| Project | What it is | How this differs |
|---|---|---|
| vLLM, SGLang, llama.cpp, Ollama, mlx-lm | The engines' built-in parsers; the ground truth this library was derived from and tested against | Importable without the engine; one state machine for streaming and non-streaming; explicit policies |
| [transformers response templates](https://huggingface.co/docs/transformers/main/chat_response_parsing) | Declarative, model-shipped output parsing with a streaming parser | No transformers dependency; works when the model ships no template; arguments stream as valid JSON. Likely the long-term standard — see above |
| [openai-harmony](https://pypi.org/project/openai-harmony/) | Official renderer and token-level parser for the Harmony format | Harmony only; use it for gpt-oss when you have token ids |
| [llm-router-utils](https://pypi.org/project/llm-router-utils/) | SGLang's reasoning and tool-call parsers and chat templates, extracted for routers; ~44 detectors | Far more formats than here. Depends on pinned `transformers`, `openai`, `fastapi`, `xgrammar` (→ `torch`) |
| [tool-parser](https://docs.rs/crate/tool-parser) (Rust crate) | Streaming tool-call parsers for about a dozen formats, Apache-2.0 | Rust. The PyPI package of the same name is a 0.0.1 placeholder announcing Python bindings |
| [Tooletta](https://pypi.org/project/tooletta/) | Parse and render tool calls between dialects for training and distillation data | Complete strings, not streams; rendering is its focus, which this library does not do |
| [iyulab/ToolCallParser](https://github.com/iyulab/ToolCallParser) | .NET library normalising tool calls across provider APIs, plus Hermes/Qwen text markup | .NET; provider JSON rather than raw model text; streaming is limited to complete calls per chunk |
| [partial-json-parser](https://pypi.org/project/partial-json-parser/), [json-repair](https://pypi.org/project/json-repair/) | Parse incomplete / repair broken JSON | JSON only, re-parse per call; here the transcoder emits incremental fragments and knows the surrounding dialect |

Projects that were looked for and not found under the names they were mentioned by (so nothing is claimed about them): a package or repository called `toolcall-dialects`, and the `shepherd-ai/shepherd` repository that the PyPI `tool-parser` placeholder links to.

## FAQ

**I use vLLM / SGLang / Ollama / llama-server. Do I need this?**
No. Their servers parse for you. This is for code that sits where those servers would: your own endpoint, a proxy in front of a raw completion API, batch post-processing, evals.

**Which dialect is my model?**
`detect(chat_template=tokenizer.chat_template)` — the template contains the markers the model was trained on. `llm_dialects.dialects()` lists model families. For GGUF files the template is in the metadata (`tokenizer.chat_template`).

**Why are the deltas different when I change the chunk size?**
The *text* of deltas follows the chunks; their concatenation does not. Compare `coalesce(events)` or use `parse()`.

**Why did whitespace disappear?**
Whitespace that touches a marker (`</think>\n\n`, `\n<tool_call>`) is treated as markup, uniformly for all dialects. Everything else is verbatim. [docs/design.md](docs/design.md#whitespace)

**My model's `reasoning_content` is empty and the answer starts with thinking text.**
The chat template opened `<think>` in the prompt. Use `reasoning="prefilled"` (or `detect()`). The symptom is a `stray_reasoning_close` issue.

**Arguments arrive in one piece instead of streaming.**
For GLM, MiniMax-M2 and Qwen3-Coder, a value's JSON type is unknown until it ends unless you pass `tools=`; then string parameters stream.

**Can it constrain generation so the output is always valid?**
No — that needs the sampler (xgrammar, llguidance, GBNF). This library deals with whatever was generated.

**Can it render tool calls and tool results back into the prompt?**
No. That is the chat template's job.

**Is the default policy safe for agents?**
It is safe in the sense that it never crashes and never hides anything: check `call.status` and `call.known` before executing. For "refuse anything irregular", use `STRICT`.

## Roadmap

- Real-output corpus cases for the dialects that have only engine vectors (needs larger models than fit the current test setup: Mistral Small, Phi-4-mini, Gemma 4, Granite 3.x, Qwen3-Coder).
- More dialects, in order of demand: Cohere Command, LFM2, DeepSeek V3.2/V4 (DSML), Kimi K3, MiniMax-M3, InternLM, xLAM, Granite 3.2 reasoning.
- Load a Hugging Face `response_template` as a dialect.
- Optional token-id input, to tell special tokens from look-alike text.
- OpenAI Responses API adapters.
- PyPI release once the API has settled with real users.

## Contributing

Issues with raw model output are the best contribution; there is a "New dialect / wrong parse" form for them. For code:

```bash
pip install -e ".[dev]"
ruff check . && ruff format --check . && mypy && pytest
```

Adding a dialect requires cited sources (engine code at a commit and the model's chat template — not memory), corpus cases with honest provenance, and a writer in `tests/renderers.py` so the round-trip and chunk-boundary properties cover it. [CONTRIBUTING.md](CONTRIBUTING.md) has the checklist.

## License

Apache-2.0. Copyright 2026 The llm-dialects Authors. Test vectors adapted from vLLM, SGLang, llama.cpp and Ollama are attributed in [NOTICE](NOTICE).
