Metadata-Version: 2.4
Name: kamiwaza-mlx
Version: 0.2.9
Summary: Unified MLX server & CLI (language and vision) with OpenAI-compatible endpoints
Author-email: Matt <opensource@kamiwaza.ai>
License: MIT License
Project-URL: Homepage, https://github.com/kamiwaza-ai/kamiwaza-mlx
Project-URL: Bug Tracker, https://github.com/kamiwaza-ai/kamiwaza-mlx/issues
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: certifi>=2026.10.19
Requires-Dist: fastapi>=0.128.3
Requires-Dist: mlx==0.32.3
Requires-Dist: mlx-lm==0.31.3
Requires-Dist: mlx-metal==0.32.3
Requires-Dist: mlx-vlm==0.7.4
Requires-Dist: mlx-audio==0.5.7
Requires-Dist: transformers<5.17.0,>=5.14.0
Requires-Dist: huggingface-hub<2.0.0,>=1.0.0
Requires-Dist: llguidance<2.0.0,>=1.7.5
Requires-Dist: pillow>=12.1.0
Requires-Dist: pydantic>=2.12.5
Requires-Dist: requests>=2.32.5
Requires-Dist: sentence-transformers>=5.2.2
Requires-Dist: torchvision>=0.25.0
Requires-Dist: uvicorn>=0.40.0

# Kamiwaza-MLX 📦

A simple openai (chat.completions) compatible mlx server that:
- Supports both vision models (via flag or declared config detection) and text-only models
- Supports streaming boolean flag
- Has a --strip-thinking which will remove <think></think> tag (in both streaming and not) - good for backwards compat
- Supports usage to the client in openai style
- Prints usage on the server side output
- Appears to deliver reasonably good performance across all paths (streaming/not, vision/not)
- Has a terminal client that works with the server, which also support syntax like `image:/path/to/image.png Describe this image in detail`
- Experimental direct multi-node bootstrap via `mlx.distributed` when `PAIRED_HOST` is provided; the current Kamiwaza TP=2 launcher/proxy flow is documented in `MLX_TWO_NODE_README.md`

Tested largely with Qwen2.5-VL and Qwen3 models.

## Current MLX Stack

The Desktop candidate branch locks the MLX compatibility epoch as one
dependency unit. Dependency/import checks and offline regressions pass; semantic
vision and distributed runtime qualification remain open. These versions must move together because current `mlx-vlm` requires MLX
0.32 while `mlx-lm` remains on its 0.31 release line:

- `mlx==0.32.3`
- `mlx-lm==0.31.3`
- `mlx-metal==0.32.3`
- `mlx-vlm==0.7.4`
- `mlx-audio==0.5.7`

Vision capability is fail-closed. If a model config declares a vision tower,
the server loads it through `mlx-vlm` or fails startup with the underlying
diagnostic; it never silently falls back to a language-only `mlx-lm` process.
Repository names alone do not declare vision capability. Use `--text-only`
for an intentionally text-only conversion that retains vision config metadata;
it is mutually exclusive with `--vision`. Requesting DFlash also remains an
explicit text-only mode and is logged as such for vision-capable checkpoints.

Earlier-epoch testing of `mlx-community/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-4bit`
used `mlx_vlm generate` for combined text, image, and audio inputs. This model
has not been requalified against the current lock. Native video is
not yet functional in this MLX path; the current upstream Nemotron MLX model
code raises `NotImplementedError: Efficient video sampling is not implemented
for Nemotron Omni yet.`

## Checkpoint code on the MLX runtime

On the MLX runtime of `kamiwaza-mlx-server`, **`--trust-remote-code` is off
by default**, including when this entry point runs on multiple ranks. The flag
allows checkpoint and processor-requested Python when loading the target and
any drafter. Their loaders may also request code from another repository.
Each loader resolves Hub IDs as it did before this change. With trust off,
the launcher checks the resulting directory before passing it to the loader.
Identical Hub IDs share a resolution when they use the same resolver. A
fallback from text to vision uses mlx-vlm's resolver and vision check. If
each resolver returns a config that selects the other loader, startup asks
for the local directory of the intended revision.

Run `python -m kamiwaza_mlx.server` from a trusted working directory, never
from inside a model directory. Python puts the working directory on the import
path by default for `-m`. The installed `kamiwaza-mlx-server` console script
uses its script directory for that import path entry.

| Path | Refused before load with trust off | Why | Measured (offline stand-ins, pinned packages) |
|---|---|---|---|
| Text target (mlx-lm), including `--text-only` | `model_file` in `config.json` | mlx-lm imports the named file even with trust off. A config `auto_map` passes the directory check. A built-in tokenizer can still load when Transformers declines a custom `AutoConfig`. | In 10 noninteractive `llama` attempts with the trust argument omitted, no `auto_map` code ran. The loader imported a declared `model_file`. |
| Vision target (mlx-vlm), including a fallback from text | `model_file` in `config.json`, a non-empty `auto_map`, `audio_tokenizer_class` or `audio_tokenizer_name_or_path`, or a model type listed below | mlx-vlm can import the model file or delegate tokenizer and processor loading. The launcher checks before each call to the vision loader. | On `qwen4_exp`, `auto_map` code ran in 8 of 12 attempts with the trust argument omitted and 0 of 12 with `trust_remote_code=False`. Both settings imported a declared `model_file`. On `qwen3_vl`, tokenizer and config `auto_map` code ran with the weights step stubbed. |
| Drafter (`--draft-model` or `--dflash-draft-model`) | `model_file` in `config.json` | Two drafter loaders import the named file. A drafter's `auto_map` alone does not trigger this check. The drafter check runs before the target loads. | In 21 loads (seven per backend), dflash-mlx and mlx-vlm `load_drafter` imported `model_file`. Z-Lab imported no checkpoint module with Hub lookup redirected to the fixture. None of the three imported `auto_map` code. |

With trust off, all three paths check for `model_file` at any depth in the
`config.json` the loader opens, including spellings the filesystem treats
as equivalent. An unreadable config stops startup, and a drafter directory
must contain `config.json`. Vision also checks for `auto_map` and the two audio
tokenizer keys at any depth in each top-level JSON file. The vision check
applies its canonical file rules to its fixed list of loader filenames and
the config Transformers selects from `configuration_files` in `config.json`.
It refuses dangling links and requires each present canonical file to be
readable JSON of at most 64 MiB. It requires a JSON object except for
`engram_token_map.json`, which may have any JSON root type. The size limit,
JSON syntax check, and vision declaration scan still apply. For other top-level
JSON files the vision check skips unreadable or malformed files, warns for
oversized files, and refuses excessive nesting. A Python file that nothing
names can remain in any directory.

On the text path, a tokenizer `auto_map` also passes the directory check.
The launcher passes the explicit trust setting to mlx-lm's tokenizer and to
mlx-vlm's model and native drafter loaders. Opting in passes `True` and can
load a custom text tokenizer without prompting. Processor-only declarations
do not trigger the text directory check. With trust off, the Transformers guard refuses
remote-only classes and direct dynamic class imports, even if a dependency
asks for `trust_remote_code=True`. A loader that recovers using built-in code
can finish loading.

On refusal, the launcher exits 2 and attempts to write one error line. A
declaration refusal lists the file and matching fields, shortening long entries
and counting additional matches when the list is full. A Transformers
refusal names the model path or ID. With trust off, the launcher reports the
file and `auto_map` entry when it recognizes the declaration involved in a
text tokenizer failure.
Both kinds of refusal identify `--trust-remote-code`. An inspection error
names the path and the reason inspection failed.

The vision check refuses the following canonical types in mlx-vlm 0.7.4.
Their processors have paths that request or default to trusted tokenizer
loading. The check uses the top-level `model_type` when truthy, otherwise
`speculators_model_type`, then lowercases and applies mlx-vlm's remapping.

| Canonical type | Processor behavior |
|---|---|
| `got`, `inkling` (including `inkling_mm_model`) | Requests trust for its tokenizer. |
| `internvl_chat` | Requests trust when given a tokenizer name. |
| `minicpmo`, `minicpmv4_6` | After a built-in tokenizer failure, takes the trusted fallback only when trust is enabled. |
| `moondream2` (including `moondream1`) | Loads a tokenizer from another repository with trust enabled. |
| `moondream3` | Can fall back to another repository's tokenizer. Preserves explicit trust and defaults to true when the argument is omitted. |

The original offline probe recorded 560 cases across MLX and oMLX, comprising
538 model or engine load attempts and 22 standalone tokenizer loads. The
stand-in modules recorded import stacks and supplied minimal loader classes.
The 21 drafter loads were measured separately.

Five of the 45 fetched catalog configs declare a non-empty `auto_map`, including
`mlx-community/MiniMax-M2.7-4bit` and `mlx-community/DeepSeek-R1-0528-4bit`.
Core lists both conversions as MLX variants. Their `auto_map` declarations
pass the default mlx-lm directory check.
None of the 45 fetched configs declares `model_file`.
Config fetches failed for three catalog repositories,
`cognitivecomputations/Qwen3-235B-A22B-Thinking-2507-AWQ`,
`koushd/Qwen3-235B-A22B-Instruct-2507-AWQ` and `QuixiAI/Kimi-K2-AWQ`.

These checks assume other processes running as the same user leave checkpoint
directories and files alone during startup. Some mlx-vlm processors render
checkpoint chat templates outside a sandbox. That behavior was identified by
source inspection and remains open under ENG-18520. The two-node launcher
(`kamiwaza-mlx-distributed`) and `kamiwaza-mlx-distributed-dflash-server`
retain their existing behavior. Neither distributed command was measured.
oMLX uses its own `trust_remote_code` model setting.

## Choosing the runtime: MLX or oMLX

`kamiwaza-mlx-server` can serve a model itself (mlx-lm / mlx-vlm in this
process) or hand it to [oMLX](https://github.com/jundot/omlx) (Apache-2.0),
which runs from its own environment.

```bash
kamiwaza-mlx-server -m <model dir>                  # --runtime auto (default)
kamiwaza-mlx-server -m <model dir> --runtime mlx    # this server, always
kamiwaza-mlx-server -m <model dir> --runtime omlx   # oMLX, always
```

`auto` reads `model_type` from the model's `config.json`. The shipped
`kamiwaza_mlx/config/runtimes.json` gives the following choices.

| `model_type` | Model | `auto` picks |
|---|---|---|
| `deepseek_v41` | DeepSeek V4.1, including Flash | oMLX |
| `qwen4_exp` | Qwen3.8-Flash-Next | oMLX |
| A model that auto would otherwise serve with oMLX, launched with an option that selects MLX, such as `--trust-remote-code`, `--ple-store`, `--draft-model`, `--dflash-draft-model` or `--distributed` (see "Options of the MLX runtime on oMLX" for flags and environment variables) | | MLX. The runtime line names the first matching MLX-only option or the environment variable that decided the choice, for example `runtime=mlx (auto: model_type=qwen4_exp, --ple-store given, mlx only)` |
| `qwen4_exp` whose `config.json` declares `text_config.ple_storage` or `llm_config.ple_storage` (the mlx-vlm external-table layout, our `-extple` directories) | | MLX: oMLX cannot load that layout. `--runtime omlx` on it is refused |
| `inkling2` | Inkling 2 Small | oMLX (refused for now, see below) |
| anything else, or no local `config.json` | | MLX |

The table is data. To change a default, edit that file or point
`KAMIWAZA_MLX_RUNTIME_TABLE` at your own copy. `KAMIWAZA_MLX_RUNTIME` sets the
option from the environment. The command line wins. The runtime line says
which runtime the launcher chose and why.

The launcher never falls back from one runtime to the other after a
failure. If oMLX is selected and cannot be used, it exits with status 2 and
prints the reason and the fix. When an option or checkpoint layout selects
MLX, the runtime line names it before the model loads.

**Upgrading from a build with fixed sampling defaults.** On the MLX runtime,
each omitted or null sampling field now defaults to its valid checkpoint
value when available, even if the request sets the other fields. For example, a request
sending only `temperature` to Qwen3.8-Flash-Next now uses `top_p=0.95` and
`top_k=20` when those values are in the loaded checkpoint's
`generation_config.json`. To use the old defaults, start the server with
`--sampling-defaults launcher` or send
`{"temperature": 1.0, "top_p": 1.0, "top_k": 0}` in the request.
Earlier builds ignored the request's `top_k`. The launcher now applies it
on backends that support top-k. For a non-null `top_k`, the server rejects
negative integers, booleans and other non-integers. Requests with `-1`,
`true` or `20.0` return HTTP 422. Some servers accept `-1` for no limit.
Send `0` for no top-k limit here.

**Upgrading from a build without vision prompt reuse.** On the MLX runtime,
a `qwen4_exp` vision-language model can now reuse matching prompt prefixes
by default when KV caching is enabled. The launcher checks reuse after the
model has loaded and any native vision drafter has passed its startup
self-test. Only MTP or no drafter qualifies. With `--vlm-prompt-reuse on`,
startup requires a working checkpoint manager. A checkpoint can still exceed
the memory cap. The default can keep up to two checkpoints whose cache buffers
total at most 8 GiB, and each active restore needs memory for its own copy.
Requests with images or media placeholder strings in the rendered prompt
read the full prompt. Use `--vlm-prompt-reuse off` for a full read on every
vision request.

On development commit `9174b27`, the Qwen3.8-Flash-Next assistant messages
matched with reuse on and off in one Mac Studio run at temperature 0, both
without a drafter and with the MTP drafter. The comparison left out the
tool-call IDs the server generates per response. The measurements are under
"Qwen3.8-Flash-Next with a drafter".

**Upgrading from 0.2.7 or earlier.** `--runtime` is new in 0.2.9, the first
release published after 0.2.7 (0.2.8 was a version number used during
development and was not published). Up to 0.2.7 the launcher used only MLX.
With `--runtime auto`, a local checkpoint identified as DeepSeek V4.1 or
Qwen3.8-Flash-Next now goes to oMLX unless its layout or the launch options
select MLX. For a hub ID, `auto` starts with the table default (MLX in the
shipped table).
A host without the oMLX environment refuses an oMLX launch (exit 2,
with the install command) until the environment is installed or the launch
says `--runtime mlx`. A launch that passes other options of this server
(KV cache, tool parser, tracing and so on) for one of these models is refused
on oMLX too, naming the option; add `--runtime mlx` to keep it on this
server. Abbreviated options (`--mod` for `--model`, for example) are no longer
accepted by this server on either runtime; spell options out.

The new `--draft-model` option enables a native mlx-vlm drafter.
An explicit `--draft-block-size` must be at least 2 because one token comes
from the target and the rest are drafted. Startup requires a completed draft
round in up to three greedy text generations of up to eight tokens each.
The launch logs the round count on success and exits 2 if generation raises
or none of the three prompts runs a round.
On oMLX, a Qwen3.8-Flash-Next launch with a truthy `mtp_enabled` setting
requires an embedded draft-head tensor name in the checkpoint metadata.
On a launch that serves on the MLX runtime, `kamiwaza-mlx-server` reads its
bundled `config/reasoning.json` before it loads the model and exits 2 on a
file-read error, a JSON decoding error or a failed check of the table. A
launch that goes to oMLX does not read it.

Chat requests on the MLX runtime have these changes (see also "Reasoning controls").

- With a native drafter, an unset, null, zero or negative token limit uses
  the served context window minus the prompt text token count, at least 1,
  or `2**31 - 1` if the window is unknown. Image tokens are excluded, so an
  image request can get more room than remains in the window. The vision
  path reports `stop` for an ordinary answer that reaches this bound.
  Generation keeps its previous limits for positive values or a launch
  without a native drafter. With a native drafter the server treats zero as
  unset. On a vision-language launch without one it passes zero through to
  mlx-vlm; for an autoregressive model its generation then emits no tokens.
- The `kamiwaza-mlx-server` chat endpoint on the MLX runtime now accepts
  `max_completion_tokens` as an alias for `max_tokens`. Version 0.2.7
  ignored it. A null for either name now means unset, though
  `max_tokens: null` gave a `422` in 0.2.7.
  Two non-null values must match or the server returns a `400`.
- 0.2.7 ignored `reasoning_effort` and `chat_template_kwargs`. On a launch
  that passes them to the chat template, the server now passes the merged
  arguments through unless it has recorded reasoning facts for the model.
  A native vision-language launch adds its thinking-off default only when
  neither reasoning field is set. On Qwen3.8-Flash-Next, `low`, `medium`
  and `xhigh` sent alone turn reasoning on at that level. `minimal` is sent
  as `low`, `high` as `xhigh`, and `none` turns reasoning off.
  On the Qwen native vision-language path, `enable_thinking: true` sent
  alone now reaches the template.
- With stripping on and auto-detection on, whether an answer starts inside
  a think block is now decided per request from the think tag that ends
  its rendered prompt, on every path (the rule is under "Qwen3.8-Flash-Next
  with a drafter"). Visible effects: a request rendered with thinking off
  (as on the default Qwen3.8-Flash-Next MLX launch) keeps a short answer
  that could come back empty before; a vision-language stream without tools
  rendered with thinking on no longer sends the reasoning and `</think>` in
  `content` (a block of more than 50 tokens with no think tag that a token
  limit ends is still returned as the answer); a short output with no think tag, in a block inferred from the
  prompt, is withheld on every path, including one-piece answers and
  streams that used to send it; and a stripped vision-language stream that
  sees a `</think>` outside a think block now updates the process-wide
  detection result, as the other paths already did. Text and DFlash
  streams now keep visible text that comes before an opening tag in the
  same piece, and streams without tools recognize a closing tag split
  across pieces inside a block. Streams without tools withhold a block whose opening tag they
  recognized and that does not close; text and DFlash streams used to send
  a long one at the end. Vision-language streams match think tags in any
  case and remove a `</think>` that has no opening tag when it arrives in
  one piece. Think tags are now found in the original text, so a character
  whose lower case is longer (for example `İ`) no longer shifts the split;
  this applies with stripping or auto-detection off too.
- Contradictory fields, reserved template argument names and unrecognized
  Qwen3.8-Flash-Next efforts give a `400`. Nested `enable_thinking` and
  `reasoning_effort` are allowed even when the renderer names them as
  parameters. When the fields are consistent and otherwise valid, the
  unrecognized-effort error lists the accepted names and mappings. When a
  comparison render attributes an error to the arguments, the `400` quotes
  the template error. In 0.2.7 a text renderer's `TypeError` could cause a
  retry without `enable_thinking`.
  Request validation can give a `422` before these checks.
- A text renderer's `TypeError`, `ValueError` or Jinja template error that
  remains after tool fallback and field checks now gives a `503` quoting
  the error in place of an opaque `500`. A `503` from the native template
  renderer also quotes its error.
- A launch that uses manual text formatting or mlx-vlm's non-native prompt
  helper refuses reasoning fields with a `400`. This includes
  `enable_thinking`, which 0.2.7 accepted there and ignored.

For an unforced `minimax-m2` request on a text template that takes tools,
the server renders the same request with a unique name for each tool.
It uses the original prompt only if the second render succeeds, contains
every marker and matches the original exactly after restoring the real names.
In the renamed render each name must be the name of a JSON declaration
inside its own `<tool>...</tool>` block.
Otherwise it renders explicit instructions with the request's own arguments.
The parser reads only MiniMax's call format, so a template that presents
tools another way (Qwen3-Coder's, for example) gets explicit instructions.
A tools-only comparison that raises `TypeError` also uses this fallback.
MiniMax always uses explicit instructions for a forced choice, even on a
template that prints `<tool>` blocks.
Explicit instructions list every supplied tool in request order, even for a
forced choice. A missing declaration or missing instructions in the
rendered prompt give a `400`. On these
explicit-instruction paths, a forced `tool_choice` naming a tool absent
from the request's `tools` is also a `400`. With `--tool-call-parser openai-json`,
a forced choice now lists every supplied tool before the forced instruction.
Previously it showed only the selected function's schema. With tools enabled,
`openai-json` now returns a `400` for an otherwise valid tool with a `function`
member but a type other than `function`, with or without a forced choice.
In 0.2.7, `openai-json` could silently omit non-function tools.

**From a 0.2.8 development build to 0.2.9.** These changes are relative
to the last 0.2.8 development snapshot (`fef2c29`).

- `kamiwaza-mlx-distributed` and `kamiwaza-mlx-distributed-dflash-server`
  now exit 2 when `KAMIWAZA_MLX_RUNTIME` is set to something that is not
  `auto`, `mlx` or `omlx`. 0.2.8 ignored such a value there.
- A single-node launch given `--distributed-world-size 1` (or given
  `--distributed-host` or `--distributed-world-size` while the environment
  turns distributed serving off) has its port checked before the model is
  loaded, like any other single-node launch. In 0.2.8 a taken port was
  refused for such a launch only after the load.
- `kamiwaza-mlx-server --help` prints the option list beside a settings file
  that cannot be read. 0.2.8 exited 2 without it.
- If the null device cannot be used when the server ends, the non-blocking
  flag is left on its output pipe (see "Tying the server to its supervisor").
  In 0.2.8 the server could hang at exit in that case.
- On a launch that serves on the MLX runtime, `kamiwaza-mlx-server` reads its
  bundled `config/reasoning.json` before it loads the model and exits 2 on a
  file-read error, a JSON decoding error or a failed check of the table. A
  launch that goes to oMLX does not read it.
- With a native MLX drafter, omitting the limit no longer cuts the answer
  to one token.
  An unset, null, zero or negative limit uses the served context window
  minus the prompt text token count, at least 1, or `2**31 - 1` if the
  window is unknown. Image tokens are excluded, so
  an image request can get more room than remains in the window. The
  vision path still reports `stop` for an ordinary answer that reaches this
  bound. Generation keeps its previous limits for positive values or a
  launch without a native drafter.
  With a native drafter the server treats zero as unset. On a vision-language
  launch without one it passes zero through to mlx-vlm; for an autoregressive
  model its generation then emits no tokens.
- The `kamiwaza-mlx-server` chat endpoint on the MLX runtime now accepts
  `max_completion_tokens` as an alias for `max_tokens`. A null for either
  name now means unset, though `max_tokens: null` gave a `422` in 0.2.8.
  Two different non-null values now give a `400` where 0.2.8 ignored
  `max_completion_tokens`.
- On `kamiwaza-mlx-server`, `reasoning_effort` and `chat_template_kwargs` now
  reach the model's chat template where the launch supports them. A native
  vision-language launch adds its thinking-off default only when neither
  reasoning field is set. On Qwen3.8-Flash-Next, `low`, `medium` and `xhigh`
  sent alone turn reasoning on at that level. `minimal` is sent as `low`,
  `high` as `xhigh`, and `none` turns reasoning off.
- With stripping on and auto-detection on, whether an answer starts inside
  a think block is now decided per request from the think tag that ends
  its rendered prompt, on every path (the rule is under "Qwen3.8-Flash-Next
  with a drafter"). Visible effects: a request rendered with thinking off
  (as on the default Qwen3.8-Flash-Next MLX launch) keeps a short answer
  that could come back empty before; a vision-language stream without tools
  rendered with thinking on no longer sends the reasoning and `</think>` in
  `content` (a block of more than 50 tokens with no think tag that a token
  limit ends is still returned as the answer); a short output with no think tag, in a block inferred from the
  prompt, is withheld on every path, including one-piece answers and
  streams that used to send it; and a stripped vision-language stream that
  sees a `</think>` outside a think block now updates the process-wide
  detection result, as the other paths already did. Text and DFlash
  streams now keep visible text that comes before an opening tag in the
  same piece, and streams without tools recognize a closing tag split
  across pieces inside a block. Streams without tools withhold a block whose opening tag they
  recognized and that does not close; text and DFlash streams used to send
  a long one at the end. Vision-language streams match think tags in any
  case and remove a `</think>` that has no opening tag when it arrives in
  one piece. Think tags are now found in the original text, so a character
  whose lower case is longer (for example `İ`) no longer shifts the split;
  this applies with stripping or auto-detection off too.
- Contradictory `enable_thinking` and `reasoning_effort` values now give a `400`.
- A reserved name in `chat_template_kwargs` on a template-backed launch now gives
  a `400`. Nested `enable_thinking` and `reasoning_effort` are allowed even
  when the renderer names them as parameters.
- An otherwise valid Qwen3.8-Flash-Next request on a template-backed launch
  with an unrecognized effort now gets a `400` listing the accepted names and mappings.
- A launch that cannot pass `enable_thinking`, `reasoning_effort` or
  `chat_template_kwargs` to a template now gives a `400` if any is set.
- A text renderer's `TypeError` could previously drop `enable_thinking` on
  retry. When the request's own render fails and a comparison render
  without the request's arguments succeeds, the request now gets a `400`
  quoting the error.
- A text renderer's `TypeError`, `ValueError` or Jinja template error that
  remains after tool fallback and field checks now gives a `503` quoting
  the error in place of an opaque `500`. A `503` from the native template
  renderer now quotes its error too.
- For an unforced `minimax-m2` request on a text template that takes tools,
  the server tries a second render with a unique name for each tool.
  It uses the original prompt only if that render succeeds, contains every
  marker and matches the original exactly after restoring the real names.
  In the renamed render each name must be the name of a JSON declaration
  inside its own `<tool>...</tool>` block.
  Otherwise it renders explicit instructions with the request's own arguments.
  The parser reads only MiniMax's call format, so a template that presents
  tools another way (Qwen3-Coder's, for example) gets explicit instructions.
  A tools-only comparison that raises `TypeError` also uses this fallback.
  A forced MiniMax choice now lists every supplied tool in request order
  before the explicit instruction to call the selected function, even on a
  template that prints `<tool>` blocks.
  The server refuses instructions that omit a supplied tool or disappear
  from the rendered prompt. On explicit-instruction paths, a forced `tool_choice`
  naming a tool absent from the request's `tools` now gets a `400`. The 0.2.8
  build could answer with tools missing. With `--tool-call-parser openai-json`,
  a forced choice now lists every supplied tool before the forced instruction.
  Previously it showed only the selected function's schema. With tools enabled,
  `openai-json` now returns a `400` for an otherwise valid tool with a `function`
  member but a type other than `function`, with or without a forced choice.
- A `--draft-model` launch now exits 2 if a greedy text self-test raises
  or all three test generations of up to eight tokens complete without a
  draft round. A successful self-test logs the number of completed rounds.
- A `--draft-model` launch now refuses `--draft-block-size 1` with exit 2
  because mlx-vlm would draft nothing and end every answer after its first token.
- With a truthy `mtp_enabled` setting, an oMLX launch of Qwen3.8-Flash-Next
  now exits 2 if the checkpoint metadata lists no embedded draft-head
  tensor. The 0.2.8 launcher did not check the metadata.

### The oMLX environment

oMLX is not a dependency of this package. It requires an exact `mlx` build
that differs from this package's, and it is published as release wheels on
GitHub, not on PyPI. Install it once per host, beside this environment:

```bash
python -m kamiwaza_mlx.omlx_backend install --prefix <root>/omlx-venv   # needs uv
python -m kamiwaza_mlx.omlx_backend check
```

`install` downloads the pinned oMLX release wheel, verifies its SHA-256, and
then installs it with `uv` together with the locked dependency set shipped in
this package: 108 packages from PyPI at exact versions and four built
from pinned git commits. Only the oMLX wheel is checked against a hash; the
other packages are pinned by version or commit, not by hash. It needs network
access and refuses to touch an existing environment. An installation that
fails part-way (a network error while the dependencies are fetched, for
example) can be finished by running the same `install` command again: the
installer marks the directory when it begins and when it has finished, and a
directory it began and did not finish is installed into again. Until then
`check` and every launch refuse that environment as incomplete, by name.
Nothing is deleted by the installer; a directory it did not begin is left as
it is. The pin (version, wheel URL, hash,
lock file) is in `kamiwaza_mlx/config/omlx.json`; today it is **oMLX 0.7.3**
with `mlx==0.32.4` on Python 3.12. The launcher looks for the environment in
`KAMIWAZA_OMLX_HOME`, or `omlx-venv` next to the environment this package is
installed in, and refuses any other oMLX version.

**Inkling 2 Small** (`inkling2`) needs oMLX support that is not in any
published release yet (upstream commit `adc3daa6`, PR #5229). Until the pin
moves to a release that has it, the launcher exits and says so. The regular
MLX runtime cannot serve it either: mlx-lm and mlx-vlm at this package's pins
have no `inkling2` model type.

### What a launch on oMLX does

- Everything oMLX reads or writes is under one state directory per model:
  `KAMIWAZA_MLX_OMLX_STATE_DIR/<model directory name>-<key>` (default
  `~/.kamiwaza-mlx/omlx/...`), where the key is derived from the model
  directory's full path, so two models whose directories share a name do not
  share state, and the same model finds its cache again. One server at a time
  uses a state directory: a second launch of the same model directory while
  the first is running exits 2; give it its own `KAMIWAZA_MLX_OMLX_STATE_DIR`
  to run two copies. It holds the oMLX profile, a view of the model
  made of links (no weights are copied), and the prompt-prefix cache. Nothing
  is written to `~/.omlx`, and `OMLX_*` environment variables are not passed
  on.
- oMLX's start-up clean-up of distributed rank processes is switched off.
  Left on, it terminates ranks recorded under `~/.omlx` whose coordinator has
  exited, and those can belong to another oMLX installation on the same Mac.
- oMLX's own `/health` answers without a key and includes the model's name
  and memory figures; the MLX runtime's `/health` returns the status only.
- Only inference routes are served (`/health`, `/v1/models`,
  `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/messages`,
  `/v1/messages/count_tokens`, `/v1/responses` and `/v1/responses/<id>`,
  `/v1/tokenize`, `/v1/detokenize`). oMLX's administration
  pages and API answer 404, and its WebSocket routes (realtime audio) are
  closed before they are accepted.
- For DeepSeek V4.1 a request is held to the model's native context: prompt
  tokens plus the output budget plus a small reserve may not exceed
  1,048,576, or the request is answered 400 before anything is generated.
  The answer is oMLX's own overflow error (code `context_length_exceeded`,
  message opening "Prompt is too long: N tokens > 1048576 maximum context
  length"), so clients that compact a conversation on overflow do so; it
  names the field the route takes its budget from (on chat both
  spellings, `max_tokens` and `max_completion_tokens`, since oMLX's parsed
  request does not record which one was sent).
  The check uses what oMLX itself resolved: the request oMLX parsed and
  validated and the budget field its route generates with, at the point
  where each generating route (`/v1/completions`, `/v1/chat/completions`,
  `/v1/messages`, `/v1/responses`, streaming or not) checks the prompt
  against the context window. The launcher does not parse request bodies.
  At start-up it verifies that the installed oMLX still has that point and
  exits 3 naming what differs if it does not.
- No request body over `KAMIWAZA_MLX_MAX_REQUEST_BODY_MB` (default 32 MiB)
  reaches oMLX, on any model, whatever its
  content type and whether it has a `Content-Length` or is chunked: it is
  answered 413. The bytes are counted as they arrive and never buffered by
  the launcher. This runs before the key check, because the web framework
  reads a body before it checks the key.
- Browser clients (pages that call the server from JavaScript): on oMLX the
  browser origins `http://127.0.0.1:<port>` and `http://localhost:<port>` are
  allowed and no other, whatever `--host` is; there is no setting for it. The
  MLX runtime sends no cross-origin headers at all, so a browser blocks every
  cross-origin read from it. A page served from another origin therefore
  works against neither runtime directly. Non-browser clients are not
  affected.
- Requests need no API key unless one is configured (see "API key" below).
- The model is loaded at start-up. `/health` answers 503 (`"status":
  "loading"`) until it is loaded and 200 afterwards. If the load fails, the
  process prints the reason and exits with status 4; it never stays up
  without a model. While the model is loading, an inference request gets
  `503` at once, with `Retry-After: 5` and
  `{"error": {"code": "model_loading", ...}}`; it is not held until the load
  ends. This answer, like the `404` for routes that are not served, comes
  before any API-key check. Both are given by a plain ASGI wrapper that
  looks at the method and the path only and does not wrap a request it lets
  through (the body-size count above is the one thing the launcher puts
  around a request), so a streamed answer and a client that disconnects are
  oMLX's own to handle. A stop signal (SIGTERM, SIGINT) during the load ends the process
  within a second or two (status 143 or 130) without finishing the load.
- The model is served under its directory name, one request at a time; later
  requests wait.
- The shipped oMLX profile turns reasoning off for Qwen3.8-Flash-Next.
  Send `"enable_thinking": true` (or the same key in `chat_template_kwargs`)
  to turn it on. On the default MLX launch, a recognized effort level or
  alias alone also turns it on. DeepSeek V4.1 reasons unless the request
  turns it off, as in the deployment it was validated with. See "Reasoning
  controls" for the levels and the differences between runtimes.
- Drafting on oMLX uses a multi-token-prediction head stored inside the
  checkpoint, turned on with
  `KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"mtp_enabled": true}'`. For
  Qwen3.8-Flash-Next, a truthy `mtp_enabled` setting requires a tensor name
  starting with `mtp.`, `language_model.mtp.`, `model.mtp.` or
  `model.language_model.mtp.`. The launcher checks the safetensors index
  if it can read it as a JSON object, or the shard headers otherwise. If it
  finds no matching name, it exits 2. To serve without drafting, set
  `mtp_enabled` to `false`. To use a separate drafter directory, see
  "Qwen3.8-Flash-Next with a drafter".
- `/v1/models` reports the context oMLX serves, which is the model's own
  (262,144 for Qwen3.8-Flash-Next). The MLX runtime reports 131,072 for the
  same directory because that is the window it serves by default
  (`--kv-cache-max-tokens`, at most 128K unless raised). Each number is true
  for its runtime.
- The launcher process becomes the oMLX process (same PID).

| Variable | Meaning |
|---|---|
| `KAMIWAZA_OMLX_HOME` | The oMLX environment |
| `KAMIWAZA_MLX_OMLX_STATE_DIR` | Parent of the per-model state directories |
| `KAMIWAZA_MLX_OMLX_SSD_CACHE_MAX` | Size of the on-disk prefix cache (default `20GB`) |
| `KAMIWAZA_MLX_OMLX_MEMORY_CEILING_GB` | oMLX memory-guard ceiling (DeepSeek V4.1 defaults to 400, for a 512 GB Mac) |
| `KAMIWAZA_MLX_MAX_REQUEST_BODY_MB` | The largest request body passed on to oMLX, in MiB (default 32; 1 to 512). A larger body is answered 413. Text requests are far below the default; a vision model that is sent images or video as base64 may need more. 512 is oMLX's own limit, which the launcher does not change; a larger value is refused at start (exit 2) |
| `KAMIWAZA_MLX_OMLX_MODEL_SETTINGS` | JSON object merged into the oMLX model settings, e.g. `{"mtp_enabled": true}`. `is_pinned` cannot be turned off. A truthy `mtp_enabled` on Qwen3.8-Flash-Next requires a draft-head tensor name in the checkpoint metadata (exit 2 if absent) |

### API key

`KAMIWAZA_MLX_API_KEY` makes the server require a key, on **either runtime**:
every route except `GET /health` then needs `Authorization: Bearer <key>` (or
`x-api-key: <key>`), and answers `401` without it. CORS preflight (`OPTIONS`)
requests carry no key and are let through. With no key set, the
server is open, as before. The earlier name `KAMIWAZA_MLX_OMLX_API_KEY` is
still accepted; if both are set to different values the launcher exits 2.
The key can also be written in a `.env` file in the directory the server is
started from; the process environment wins over the file. That file is read
the same way on both runtimes and by the launchers that refuse a key. oMLX's
own `OMLX_API_KEY` is not used: every entry point, on either runtime, exits 2
while it is set (in the environment, in the settings file, or passed with
`--env`) and names the variable to use. The rule behind it: no entry point
starts listening without authentication while a key variable it knows about
is set. The start-up output says whether a key
is in force: on the MLX runtime `API key authentication: enabled (every route except /health).` or `API key authentication: off.`; on oMLX `kamiwaza-mlx: inference API key: required` or `kamiwaza-mlx: inference API key: off (requests are not authenticated)`. (oMLX's own `API key authentication: enabled` line, which it would print on every launch because the launcher always sets its internal management key, is reworded to `oMLX management key: set (internal; ...)`.)

A program that imports `kamiwaza_mlx.server` to serve inside its own process
is held to the same rule, and it fails closed: the key is decided from the
process environment and the settings file together, so a key in either is
enforced, and where the configuration cannot be trusted (a settings file
that cannot be read, `OMLX_API_KEY` set, two key variables that disagree,
a key in a `./.env` that a named file replaces) the import ends with
`SystemExit(2)` and no server starts. Such a program cannot turn the key
off by leaving the settings file unloaded.

The key is never logged and is not written to the launch description; on
oMLX it is in the profile's `settings.json` (mode 600). The key applies to
`kamiwaza-mlx-server`. The two-node launcher (`kamiwaza-mlx-distributed`) and
`kamiwaza-mlx-distributed-dflash-server` serve through `mlx_lm.server`, which
checks no key, so they refuse to start while the variable is set.

`GET /health` exists on both runtimes. On oMLX it answers `503` while the
model is loading and `200` once it is loaded. The MLX runtime does not listen
until its load step has returned, so there `200` means the server is up; a
large model loaded lazily still reads its weights on the first request, which
is slow.

### Tying the server to its supervisor

By default the server keeps running if the process that started it dies. A
supervisor that must never leave a model server behind passes one of:

| Option | The server stops when |
|---|---|
| `--exit-with-parent` | the process that started the launcher exits |
| `--parent-pid <pid>` | the process with that PID exits (for a supervisor that is not the direct parent, for example behind a wrapper or `nohup`) |

The contract, on both runtimes:

- The watched PID is fixed when the launcher starts and is never re-read. The
  launcher prints `kamiwaza-mlx: this server stops when process <pid> exits`.
- If that process does not exist at start, or `--exit-with-parent` is used
  when the parent is already PID 1, the launcher exits 2 and starts nothing.
- When the watched process exits, however it exits, the server prints
  `kamiwaza-mlx: the supervising process <pid> is gone; stopping this server.`
  (on oMLX always; on the MLX runtime while the web server runs, and before
  that the process ends without the line),
  sends itself SIGTERM, and if it is still running 5 seconds later exits with
  status 5. Those 5 seconds are fixed: `KAMIWAZA_MLX_STOP_GRACE_SECONDS` does
  not lengthen them, so with a supervisor gone a request in flight gets at
  most 5 seconds. During a model load it stops at once.
- The watch covers the model load and, on oMLX, lives in the process that
  remains after the launcher replaces itself (same PID as the launcher).
- It watches for the process exiting, not for a signal sent to it: a
  supervisor that is stopped (SIGSTOP) or hung still counts as alive.
- Output is best effort when the pipe it goes to is full, and no write
  waits on it for longer than a set bound (below). A supervisor usually reads the server's stdout and stderr through
  pipes; once it is gone a write to them fails, and if it stops reading, a
  blocking write waits forever once the pipe is full. So, when (and only
  when) descriptor 1 or 2 is a pipe or a socket, two processes make that
  descriptor non-blocking and replace `sys.stdout` / `sys.stderr` with a thin
  wrapper around it:

  | Process | Best-effort output |
  |---|---|
  | `kamiwaza-mlx-server` on the MLX runtime (console command or `python -m kamiwaza_mlx.server`) | yes |
  | The oMLX process the launcher becomes (`--runtime omlx`) | yes |
  | `kamiwaza-mlx-distributed` (it starts the ranks and the proxy on the same pipe and is replaced by `mlx.launch`) | no: ordinary blocking output |
  | `kamiwaza-mlx-distributed-dflash-server` (one rank among others on the same pipe) | no: ordinary blocking output |
  | A program that imports `kamiwaza_mlx.server` | no: the import replaces no stream and installs no error hook and no signal handler, and its refusals are not hard exits (see "Importing the server module" below for what it does do) |

  In those two processes a write goes straight to the descriptor. If the
  pipe is full the write waits for it for at most 50 ms in total, then drops
  the rest. For the next second no write waits: each puts what fits into
  the pipe and drops the rest. So once a write has waited its full 50 ms in
  vain, no thread waits again for a second. That does not bound a reader
  that is slow but gets there: a write that the reader takes after, say,
  30 ms waits those 30 ms and loses nothing, and so does the next one, so
  such a reader slows the server down by that much per write. No write
  raises, and `flush` does nothing. Once that second has passed, a write
  adds one line saying what was lost: `[kamiwaza-mlx: output could not be
  delivered on <stderr>: N writes (M bytes) were dropped]`, at most once
  every 5 seconds; loss right before the process exits is not reported. A
  line can be cut short where the pipe filled up, and so can that line. The
  bound for a reader is those 50 ms per write: output is complete when the
  reader takes every write within 50 ms (tested with 5,000 log lines; an
  8 MiB burst of 1 KiB lines printed as fast as possible to a reader that
  reads as fast as it can was delivered whole when measured on the
  development machine, macOS on Apple silicon; the figure depends on the
  machine's pipe size and scheduling). A reader that
  pauses longer or starts late loses what did not fit in the pipe
  meanwhile, and a single write too large to be taken within 50 ms loses
  its tail. A terminal and a regular file are left exactly as they are,
  descriptor and stream.
- Side effects of the non-blocking flag, which belongs to the pipe end and
  not to the process. While the server runs, anything else that writes
  through the same pipe end sees it: native code in the server process gets
  "would block" where it used to wait (intended), and so does a child
  process or a sibling in the same shell pipeline that shares the server's
  stdout or stderr. The reading end is not affected. Give the server its own
  pipe if something else must write to it with blocking semantics while the
  server runs.
- The flag is set late and taken back. `kamiwaza-mlx-server` sets it only
  once the runtime choice has decided that this process serves on the MLX
  runtime, and the oMLX process sets it when it starts: a launch refused by
  the runtime choice (a bad option, a setting, a key, a taken port, a
  missing oMLX environment) leaves the pipe as it was. The flag is cleared
  again, where this process set it, when the process ends by a normal
  exit, a refusal or any other exit with a status of its own (load failed,
  stop deadline, supervisor gone, start-up error), and by SIGTERM after the
  web server's own stop. Just before, the process points its descriptors 1
  and 2 at the null device, so from that moment nothing it writes reaches
  the pipe (a write to a blocking pipe that is full and unread would wait
  for good, and the interpreter still writes while it shuts down). The null
  device is opened when the process replaces its output streams, not when
  the process ends.
  If descriptor 1 or 2 cannot be pointed at it (the null device could not
  be opened, or the process has no file descriptor left), no flag is
  cleared: a process that cannot hang matters more than the flag. It is also
  not cleared where the process gets no chance to: after SIGKILL or another
  signal that ends a process by default (SIGHUP, SIGQUIT and the like),
  after a crash in native code or code that calls `os._exit` on its own,
  and after a SIGTERM during the MLX runtime's start-up (from the runtime
  choice until the server serves, the model load included), which ends the
  process at once, by design. A program that writes to the same pipe after one of
  those finds it non-blocking; a supervisor that runs further commands on a
  server's pipe can clear the flag itself (`os.set_blocking(fd, True)`).
- Several servers on one pipe. Each sets the flag if it is not set and
  clears it on leaving only if it set it; every write first sets it again
  if another server has cleared it meanwhile. So after one of them leaves,
  the pipe is blocking until the next write of one that still runs, and it
  is blocking after the last has left, in whatever order they leave. One
  window remains: a server that leaves at the very moment another is
  between that check and its write (the check is made before a write and
  again after a wait), with the pipe full and unread, lets that one write
  wait for the reader.
- Importing the server module. For a program that imports `kamiwaza_mlx.server`,
  the module body runs startup and `main_entry()` serves. Argparse handles
  usage errors. Startup refusals, including native drafter load and self-test
  failures, raise a catchable `SystemExit`. Other target-load errors and
  interrupts propagate.
  Once installed with trust off, the Transformers guard stays active in that process.
  The settings file (the one named with `--distributed-env-file`, else
  `./.env`) is loaded where it always was (when distributed settings are
  resolved, after the options have been parsed), into the environment for
  names not already set; it is not loaded early for every setting as the
  console command does, so an option default taken from the environment
  does not see it. Differences from before: the file is read by the parser
  described under "Every setting on every path" (a line that is not a setting is an
  error instead of being skipped; only matching quotes are removed; a
  leading `export` and a byte-order mark are accepted); the refusals for a
  named file that is missing or empty and for the option given twice
  apply, at the point where the key is decided, which for an importer is
  after the model has loaded; and the key is decided as
  on the console command, from the environment and the settings file
  together (the file is read in place for this, not loaded), so
  `KAMIWAZA_MLX_API_KEY` in either is enforced, an empty variable in the
  environment does not cancel a key in the file, and the refusals for
  `OMLX_API_KEY` and for a key in a `./.env` that a named file replaces
  apply. With `--exit-with-parent` or `--parent-pid` on its command line
  the importing process is watched and ended like the console command's.
  `main_entry()` itself reads no settings file.
- On the two distributed commands a full, unread pipe can still block
  ordinary output (logging, the ranks' own lines). Their refusals and
  start-up failures are armed from another thread before their line is
  written, so those exits do not depend on the pipe.
- Without either option nothing is watched; running the server from a
  terminal is unaffected.

### Exit status

| Status | Meaning |
|---|---|
| 1 | An error at start-up that no refusal covers: one line `start-up failed: ...`, then the traceback. On `kamiwaza-mlx-distributed` also a required setting that is missing or a local preparation step that reported its failure (one line; a step that failed with an error nobody handled prints the traceback too). On oMLX also a port taken between the launcher's check and oMLX's own bind: oMLX's error, before the model is loaded |
| 2 | Refused: bad option or option value, unreadable `config.json`, unusable runtime, missing, incomplete or mismatched oMLX environment, unsupported model or layout, conflicting keys, a bad setting, a settings file that cannot be read, a port that cannot be bound (for a wildcard host such as `0.0.0.0`, also a port another process holds on the loopback address alone; checked before the load for every launch on oMLX and every single-node launch on the MLX runtime, `--distributed-world-size 1` included, and not for a distributed one, where a rank that does not serve is given the same port), supervisor already gone. Nothing is served. On the MLX runtime with trust off, the launcher checks the drafter before the target loads. It checks each target path before its loader runs, including a vision fallback after a text attempt. Transformers can refuse during loading. A drafter that cannot be resolved, inspected or loaded also exits 2. Requesting `--ple-store` on a text route exits 2 before weight loading. A table supplied through `--ple-store` that mlx-vlm rejects as incompatible exits 2. If checkpoint inspection refuses the checkpoint's own PLE manifest, startup exits 2; if inspection passes and mlx-vlm rejects that manifest during loading, it is a model-load error and exits 4. A speculative drafter that loads is also refused if a test generation raises or none of three test generations completes a draft round (see "Native speculative decoding"). Drafter initialization and test-generation failures release the loaded target model. The eligibility and manager checks for `--vlm-prompt-reuse on` can also refuse the launch after the model and any vision drafter have loaded and the drafter's self-test has passed. On the MLX runtime, a port taken between the check and the web server's own bind is the same refusal, after the load (on oMLX see status 1) |
| 3 | oMLX start-up could not be made safe: a start-up guard could not be applied, the profile could not be written, or the supervising process exited in the moment between the launcher's check and oMLX starting |
| 4 | The model could not be loaded, including an ordinary vision-model load failure. MLX summarizes the error and up to four of its causes on one line and logs the traceback at debug level. oMLX reports when it finds no pinned model to load. If loading a pinned model raises an exception, it reports the exception message or, if empty, its type. |
| 5 | The supervising process exited and the server did not stop within 5 seconds of its own SIGTERM |
| 143 / 130, or ended by the signal | Stopped by SIGTERM / SIGINT. See the next table: the same stop reaches a supervisor in more than one form |

#### What a supervisor sees after a stop

The rule: a stop that the web server completes by itself ends the way the
web server ends it, and a stop that the launcher has to complete ends with
an exit status. Concretely:

- While serving, the first SIGTERM or SIGINT starts the stop deadline,
  `KAMIWAZA_MLX_STOP_GRACE_SECONDS` (default 10; 0 to 3600), on either
  runtime. If every request finishes in that time the web server stops by
  itself; if not, the process exits when the time is up with 128 plus the
  signal (143, 130), also when its output can no longer be written.
  `KAMIWAZA_MLX_OMLX_STOP_GRACE_SECONDS` is accepted as another name for the
  setting; the two set to different values are refused.
- During a model load a stop is immediate.
- When the supervising process is gone (`--exit-with-parent`,
  `--parent-pid`), the server sends itself SIGTERM, which is a stop like any
  other and so starts the stop deadline, and it exits with status 5 if it is
  still there 5 seconds later. Whichever comes first ends it: the web
  server's own stop, the stop deadline (when the grace is under 5 seconds),
  or status 5.

A supervisor that reads the wait status (Python's `Popen.returncode`, for
example) therefore sees an orderly stop and a stop by the deadline
differently, where a shell shows 143 for both. Treat "killed by SIGTERM"
and exit status 143 as the same outcome: the server stopped because it was
asked to. The cases that have been observed:

| Case | Wait status | Observed |
|---|---|---|
| SIGTERM while serving, orderly stop, either runtime | killed by SIGTERM (`returncode` -15) | live, both runtimes |
| SIGTERM while serving, a request still in flight when the grace ends | exit status 143 | live on oMLX; on the MLX runtime by test with a real web server |
| SIGINT while serving on oMLX, orderly or by the deadline | exit status 130 | live |
| SIGINT while serving on the MLX runtime, orderly stop | exit status 0 | live |
| SIGINT while serving on the MLX runtime, a request still in flight when the grace ends | exit status 130 | by test with a real web server |
| SIGTERM / SIGINT while oMLX is loading the model | exit status 143 / 130 | live |
| The model could not be loaded | exit status 4 | live |
| Refused at start | exit status 2 (3 for an oMLX start-up guard) | live |
| The supervising process was killed | gone within 0.1 to 6.1 seconds, both runtimes | live; the status has no observer left |

Not observed, from the code: SIGTERM while the MLX runtime is loading the
model ends the process killed by SIGTERM; SIGINT at any point before the MLX
runtime serves (during the load, or between the load and serving) ends it
with exit status 130.

### Every setting on every path

[docs/settings-parity.md](docs/settings-parity.md) lists the settings the package reads (a test finds them by scanning the source for environment variables read through its helpers and for the options of its parsers, and fails when one is in no row). It sets them (the API
key, distributed settings, listener, runtime choice, the MLX-only options,
oMLX settings, oMLX's own `OMLX_*` variables and the rest) against every way
it can arrive (command line, process environment, `./.env`, `--env` passed to
ranks, the selection table or an oMLX profile override) for each of the four
entry points. Every cell is honored, refused by name, or not applicable with
the reason; a variable that does not apply to an entry point is named in a
start-up line (`set but not used by ...`). The table is generated from
`kamiwaza_mlx/settings_parity.py`, and a test fails when a setting is added
without a decision.

`./.env` in the directory the server is started from is loaded into the
environment before any setting is read, on both runtimes; what is already in
the process environment wins (an empty `KAMIWAZA_MLX_API_KEY` in the
environment does not cancel a key in the file). A file named with
`--distributed-env-file` is read in place of `./.env`, as before; nothing is
merged from the two, and a start-up line says which file is in force when
both exist.

A settings file is read strictly, by one parser for all four entry points
and the distributed runtime. A line is `NAME=value`, with optional
surrounding whitespace, an optional leading `export`, and optional matching
quotes around the value; blank lines and lines that start with `#` are
skipped. Nothing after the `=` is treated as a comment: in
`KAMIWAZA_MLX_API_KEY=secret # prod` the key is `secret # prod`. Because
that is legal and seldom meant, a value that contains ` #` is pointed out at
start-up by file, line and variable name (never by value). A value that
begins with `#` (`NAME= # comment`) would be nothing but the comment: for a
key variable that is refused, naming file, line and variable; for any other
it is pointed out the same way.

Where the launcher cannot be sure what was configured, it exits 2 naming the
file and does not start: a settings file that exists and cannot be read or
decoded; a line that is not a setting (the message gives the line number and
never the line); a file named with `--distributed-env-file` that does not
exist or holds no settings, also when it names `./.env` (it does not fall
back to `./.env`); `--distributed-env-file` given more than once; and a
`./.env` that is not read because another file was named, while it holds an
API key that the named file and the environment do not.

`kamiwaza-mlx-server --help` (or `-h`) starts nothing, so it is not refused
for the settings file (`./.env`, or the one named with
`--distributed-env-file`): it prints the option list, preceded by a line
that says a launch with these settings would be refused and why.

`kamiwaza-mlx-distributed` and the DFlash distributed server do not
load `./.env`; they refuse a key or `KAMIWAZA_MLX_RUNTIME=omlx` written
there, and name any other setting found only there in a start-up line.
They also refuse a `KAMIWAZA_MLX_RUNTIME` that is not `auto`, `mlx` or
`omlx` (a typo), in the environment or in `./.env`, as `kamiwaza-mlx-server`
does: both always serve with MLX, but a misspelled value means the
configuration is not what its author meant. `kamiwaza-mlx-distributed`
refuses it in an `--env` assignment too. (Up to 0.2.7 the MLX server read that
file only for its distributed settings and, as a side effect, for anything
it happened to read later.)

### Options of the MLX runtime on oMLX

No option is dropped. Each one is either already what oMLX does, or makes
`auto` choose the MLX runtime, or stops the launch naming the option and the
runtime.

| Option | On oMLX |
|---|---|
| `-m`, `--host`, `--port`, `--exit-with-parent`, `--parent-pid` | Used |
| `--vision` | Accepted: oMLX detects a vision checkpoint itself |
| `--strip-thinking` | Accepted: oMLX returns reasoning in a separate field |
| `--trust-remote-code` | With the shipped table, the flag selects MLX when `auto` would otherwise choose oMLX. The runtime line names the first matching MLX-only option or the environment variable that decided the choice. Explicit `--runtime omlx` refuses the flag. To allow checkpoint code on oMLX, remove the flag and set `KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"trust_remote_code": true}'`. |
| `--ple-store` | With `--runtime auto`, selects the MLX runtime (see the table above). With an explicit `--runtime omlx`: refused. oMLX keeps the n-gram table of Qwen3.8-Flash-Next on disk by itself (the launcher sets `qwen4_ple_ssd_offload`), but always the table in the model's own checkpoint. Pairing weights with the table of another conversion needs `--runtime mlx` |
| `--draft-model`, `--draft-kind`, `--draft-block-size`, `--draft-queue-timeout`, `--draft-stream-send-timeout`, `--dflash-draft-model` | With `--runtime auto`, select the MLX runtime. With an explicit `--runtime omlx`: refused. oMLX drafts with the head embedded in the checkpoint: on by default for DeepSeek V4.1, and for other models through `KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"mtp_enabled": true}'`. A separate drafter directory needs the MLX runtime |
| `--vlm-prompt-reuse`, `--vlm-prompt-reuse-checkpoints`, `--vlm-prompt-reuse-max-gb` | With the packaged runtime table, these settings do not select a runtime. If `auto` chooses oMLX or you select oMLX explicitly, supplying any of these flags or a nonblank value for its environment variable stops the launch, even for `--vlm-prompt-reuse off`. oMLX manages its own prompt cache. Use `--runtime mlx` to apply these options. There, `--vlm-prompt-reuse on` also requires a model type on the checked list. |
| `--sampling-defaults` | oMLX manages its own sampling defaults. With the packaged runtime table, this option does not influence runtime selection. If `auto` or an explicit choice selects oMLX, supplying the flag or a nonblank value for its environment variable stops the launch, even for `model`. |
| `--text-only` | Refused |
| `--no-distributed` | Accepted: oMLX serves on one node |
| a word that is neither an option nor an option's value (`... 8080`) | Refused, naming it |
| `--distributed` and the other `--distributed-*` options | Under `--runtime auto` they select the MLX runtime, so the launch does not reach oMLX. With `--runtime omlx` they are refused by name |
| distributed MLX turned on through the environment or a `.env` file in the working directory (`KAMI_DIST_ENABLED`, `MLX_DISTRIBUTED_ENABLED`, `PAIRED_HOST`, `MLX_DISTRIBUTED_HOST`, `MLX_DIST_HOST`, or `PAIRED_WORLD_SIZE` / `MLX_DISTRIBUTED_WORLD_SIZE` above 1) | Under `--runtime auto` it selects the MLX runtime, named in the runtime line. With `--runtime omlx` it is refused, naming the variable: oMLX would start a separate single-node server on every rank. A distributed variable that does not turn the mode on (`RANK`, `PAIRED_PORT`, `MLX_DISTRIBUTED_TIMEOUT` and the like) has no effect on oMLX and is named in the `set but not used by ...` line |
| every other option (KV cache, tool parser, tracing, ...) | Refused: it configures this server, not oMLX |

## Sampling defaults

For stock text and vision-language generation on the MLX runtime, each of
`temperature`, `top_p` and `top_k` is chosen independently. A non-null
request value wins. For an omitted or null field, the server uses the
checkpoint's value, then falls back to `temperature=1.0`, `top_p=1.0` or
`top_k=0` (no top-k limit). An explicit zero counts as a value, as does
1.0 for `temperature` or `top_p`. In a request trace, `request` normally
holds the validated request with schema defaults before sampling defaults
are applied. Omitted sampling fields are `null`. If the server cannot
serialize the request, this field holds only `model`.
`resolved_sampling` holds the resolved values of `temperature`,
`top_p` and `top_k`.

| Flag | Environment variable | Default | Meaning |
|---|---|---|---|
| `--sampling-defaults {model,launcher}` | `KAMIWAZA_MLX_SAMPLING_DEFAULTS` | `model` | Use the checkpoint's sampling defaults. Select `launcher` to use the fixed defaults for omitted or null fields. |

In `model` mode the launcher reads `generation_config.json` from the loaded
vision model's `model_path` if it names an existing directory. Otherwise it
uses the launcher's resolved model directory. Its last choice is the loaded
tokenizer's `name_or_path` if that names an existing directory.
If no directory is known, the startup log reports `model directory unknown`
and the launcher uses its fixed defaults.

For **checkpoint values**, `temperature` must be finite and nonnegative,
`top_p` must be greater than zero and at most one, and `top_k` must be a
nonnegative integer. An invalid field produces a warning and uses its launcher
default. The startup log lists each default and its source. The launcher
ignores `do_sample` and other keys when choosing these defaults. Both pinned
model loaders also open the file. An unreadable file can prevent startup, and
mlx-lm can also reject JSON that is not an object.

For **request values**, a supplied, non-null `top_k` must be a nonnegative
integer. The request accepts numeric `temperature` and `top_p` outside the
checkpoint ranges. The pinned mlx-lm and autoregressive mlx-vlm samplers
apply top-p filtering only for `0 < top_p < 1`. Zero temperature selects
greedy decoding. With a negative temperature the sampler divides the scores
by that value, reversing their order. If a top-k or top-p filter is active,
including one taken from the checkpoint, the tokens it filtered out then
score highest. Autoregressive mlx-vlm, including Flash-Next, rounds
positive temperatures below `0.01` up to `0.01`.

For both requests and checkpoints, `top_k=0` disables the cutoff. The
launcher collects vocabulary sizes from model configuration, output weights
and tokenizer length. A cutoff at or above the largest known size also means
no limit, with a startup warning for a checkpoint value. With no known size,
the launcher passes the cutoff through. When an autoregressive sampler applies
top-k filtering, a positive cutoff at or above the actual output width raises
an error. With an EAGLE-3 drafter whose vocabulary is smaller than the
target's, mlx-vlm applies the same sampler to the draft logits, so an active
cutoff at or above the draft vocabulary size raises the same error. At zero
temperature the sampler returns the top token without applying the filter.

The `dflash-mlx` text backend discards both the temperature and the sampler,
so none of `temperature`, `top_p` or `top_k` applies there, whether supplied
by the request or the checkpoint. The `--dflash-backend auto` setting can
also select this backend when installed.
DiffusionGemma in the pinned mlx-vlm ignores `top_p` and `top_k` from both
requests and checkpoints.

## Tool-call parsing

With tools enabled on MLX, `qwen-xml` first applies its existing whole-answer
grammar to the visible text. An answer consisting of valid calls and whitespace
returns those calls in order with `message.content` set to `null`.

In mixed text, the parser takes every complete `<tool_call>…</tool_call>` block
outside reasoning, including blocks in prose or Markdown fences. It skips paired
`<think>` spans left in the visible text and everything from an unmatched opener
to the end. It skips the whole mixed answer as unfinished reasoning if the
generation prompt ends in `<think>` after trimming trailing whitespace and the
raw output has no `</think>`. Both tag checks ignore ASCII case.

Each block uses the call-only grammar to check its function name and arguments.
If a block fails to parse, or any individual gap outside reasoning
contains a wrapper tag matching `</?tool_call(?![A-Za-z0-9_-])`, the whole set
is discarded. With no calls, the server keeps the visible text unchanged and
runs its existing validation on it. Tool choice and parallel-call limits apply
to extracted calls. A tool-output error produces HTTP 502 in JSON or an
`invalid_model_tool_output` event followed by `[DONE]` in an HTTP 200 stream.

Read both `message.tool_calls` and `message.content`. Removing the blocks leaves
the original text and retained reasoning in order, with outer whitespace trimmed
and an empty result represented by `null`. A buffered stream sends reasoning,
then any remaining content in one delta, then the calls in order and the
`tool_calls` finish reason. Argument JSON uses ASCII escapes for a call if its
arguments cannot otherwise be encoded as UTF-8.

### Differences from other servers

In vLLM `v0.9.2`, the non-streaming Hermes parser returns the text before the
first call and drops text between and after calls. It accepts an unclosed final
block if the text through the end decodes to a valid Hermes call object.
Invalid JSON in any block yields the whole text with no calls. Excluding
reasoning from Hermes input requires a configured upstream reasoning parser.

## Reasoning controls

How much a model reasons is decided by its chat template. A chat request
sets the template's arguments with three fields:

| Field | Meaning |
|---|---|
| `enable_thinking` | `true` or `false`: reasoning on or off for this request |
| `reasoning_effort` | how much to reason. The Qwen3.8-Flash-Next template uses `low`, `medium` and `xhigh` (its default when thinking is on, and it can think for minutes). The MLX runtime also accepts the names listed below |
| `chat_template_kwargs` | an object of further template arguments, for example `{"enable_thinking": true, "reasoning_effort": "low"}` or `{"preserve_thinking": false}`. On the MLX runtime non-null values of the two fields above win over the same names inside it. A `null` for any of the three fields, or for `enable_thinking` / `reasoning_effort` inside the object, means "not set" |

For Qwen3.8-Flash-Next, reasoning is off on the default MLX launch and in
the shipped oMLX profile. On MLX an accepted effort name other than `none`, sent alone,
turns it on. On oMLX, send `"enable_thinking": true` (see "On oMLX" below).
The MLX runtime loads this model through mlx-vlm. The pinned mlx-lm text
loader does not take this model type.

### On the MLX runtime

These reasoning, token-limit and `max_completion_tokens` rules are those of
`kamiwaza-mlx-server`'s own chat endpoint on the MLX runtime.
`kamiwaza-mlx-distributed` and `kamiwaza-mlx-distributed-dflash-server`
serve through mlx-lm's own server, which handles these fields its own way.

The API validates the request before the server resolves its tool parser
and reasoning fields. Invalid field types can give a `422`, and an
incompatible explicit `qwen-xml` parser gives a `503`. The server checks
template argument names before checking the following combinations.

1. `enable_thinking` true with `reasoning_effort` `none` gives a `400`.
2. `enable_thinking` false with any effort other than `none` gives a `400`.
3. `enable_thinking` true and no `reasoning_effort`: an effort is added
   only for a model recorded as needing one. Qwen3.8-Flash-Next uses its
   template default, `xhigh`.
4. An effort level and no `enable_thinking`: the server turns thinking on
   only for a model recorded as needing it for the level to take effect.
   Qwen3.8-Flash-Next reads the effort only while thinking is on.

Rules 1 and 2 hold for every model. What else the launcher does depends on
what it knows about the model:

- **A model type with recorded reasoning facts.** They are in
  `kamiwaza_mlx/config/reasoning.json`, by the `model_type` in the model's
  `config.json` (for a model named by a hub id, the loaded checkpoint's),
  and exist only where a model was measured to need them.
  The server accepts the model's own levels and all six standard OpenAI API
  effort names (`none`, `minimal`, `low`, `medium`, `high` and `xhigh`).
  For Qwen3.8-Flash-Next (`qwen4_exp`), it accepts `none`
  (`enable_thinking` false and no level), `minimal` (sent as `low`), `low`,
  `medium`, `high` (sent as `xhigh`) and `xhigh`. After the switch and
  contradiction checks, an unrecognized effort gives a `400` listing the
  accepted names and mappings. A level or mapped name sent alone gets
  `enable_thinking` true added (rule 4). Thinking on alone gets nothing
  added (rule 3).
- **Any other model type** (no recorded facts, or a model whose type cannot
  be read). The launcher passes the parsed, merged fields through without
  model-specific additions, renames or effort validation. A number is passed
  through as an effort too. If either reasoning field is set, the server
  adds no default of its own. In that case the server passes the request's
  merged template arguments and adds no reasoning default. On a text launch
  that renders through mlx-lm's tokenizer wrapper, the wrapper supplies a
  missing `enable_thinking`: true if it recognizes thinking tokens in the
  tokenizer, false otherwise. A comparison render can identify a failure
  due to the request's arguments and return a `400` with the template error
  (see "Details"). The server logs pass-through once per process.
- **A launch where the fields would not reach a template.** A request that
  sets any of the three is a `400`. For otherwise valid fields, the error
  names the field and the launch. This applies to a text launch where the
  server formats messages manually, and to a vision-language model on
  a tool parser other than `qwen-xml` (another model, or `--tool-call-parser
  openai-json`, `minimax-m2` or `none`), whose prompt is built by mlx-vlm's
  prompt helper, which this server calls without template arguments.
  (mlx-vlm 0.7.4's helper could be given them; doing so is a change of its
  own and is not made in this release.) A request for no reasoning is
  refused there like any other.

| The request sends | Recorded facts (Qwen3.8-Flash-Next) | No recorded facts | No template gets the fields |
|---|---|---|---|
| none of the three | thinking off on the default vision-language launch | `enable_thinking` false is sent on the native vision-language path (a template that does not read it ignores it); the loader and template default on text | the launch's default |
| `enable_thinking` true | reasoning on at the template's default level; nothing added | to the template as sent | `400` |
| `enable_thinking` false | reasoning off | to the template as sent | `400` |
| `reasoning_effort` `low`, `medium` or `xhigh`, alone | reasoning on at that level (`enable_thinking` true added) | the server adds no default. Only the effort reaches the native vision-language template. On text, mlx-lm also sends `enable_thinking`, true for a tokenizer with thinking tokens and false otherwise | `400` |
| the same with `enable_thinking` true | reasoning on at that level | to the template as sent | `400` |
| `reasoning_effort` `minimal` or `high`, alone or with `enable_thinking` true | reasoning on at `low` or `xhigh`, respectively | the effort passes through. On text, mlx-lm supplies a missing `enable_thinking` | `400` |
| `reasoning_effort` `none`, alone or with `enable_thinking` false | reasoning off | the launcher passes the fields through. The template can accept, ignore or refuse them. To ask for thinking off, send `enable_thinking` false alone | `400` |
| any other effort (`off`, `turbo`, a number), alone or with `enable_thinking` true | `400` listing the accepted names and mappings | the effort passes through. On text, mlx-lm supplies a missing `enable_thinking` | `400` |
| `enable_thinking` true with `none`, or `enable_thinking` false with any effort other than `none` | `400` that names the contradiction | the same `400` | the same `400` |
| `chat_template_kwargs` | permitted variables go to the template. Nested reasoning controls follow the same rules as the fields | permitted variables go to the template as sent. Unrelated variables alone keep the native vision-language default | `400` if any variable is set. An empty object or one containing only null reasoning controls sets nothing. A null ordinary variable is set |

On the native vision-language path, the server adds its thinking-off
launch default (`enable_thinking` false) only when the request sets neither
`enable_thinking` nor `reasoning_effort`, at the top level or inside
`chat_template_kwargs`. A recorded effort mapping can also send
`enable_thinking` false (`none` on Qwen3.8-Flash-Next). Unrelated template
arguments keep that default. For a text model, mlx-lm 0.31.3 supplies
its loader default, true for a tokenizer with
thinking tokens and false otherwise.

Details:

- What the server cannot see. A template takes what it reads: one that has
  no `reasoning_effort` renders the same at every level, and one that has no
  `enable_thinking` renders the same either way. The server cannot tell
  whether the template used an argument its renderer accepted. The MiniMax
  check establishes that the template prints a JSON declaration for every
  supplied tool in its own `<tool>...</tool>` block, under the tool's name.
  It does not read the template's instructions for how to call a tool: a
  template that prints such declarations and asks for another call format
  passes, and its calls come back as text.
  A variable the client sends in `chat_template_kwargs` is its instruction
  to the template. Qwen probes with marked tools before the real render,
  while MiniMax probes after a successful real render. The native Qwen
  check requires every marker it adds to the names and schemas to appear
  in the probe, and each real name to occur in the real prompt in literal
  or JSON-escaped form. If the probe passes, a tool that a template
  variable removes from the real declarations can still pass the name
  check when its name occurs elsewhere in the real prompt, such as in the
  conversation, another tool's name or the forced-choice instruction.
  Other spellings some clients use (`reasoning`, `thinking`, `think`) are not
  fields of this API and are ignored like any unknown field.
- Rules 1 and 2 read `enable_thinking` as `true` or `false`. Only the effort
  `none` means no reasoning, ignoring case and surrounding whitespace.
  Recorded levels and mapped names are matched the same way. The request's own
  `enable_thinking` field is a boolean of the API on every tier: it is read
  as one when the request is parsed (a JSON `1` or `"yes"` arrives as
  `true`), before any of this applies. A non-null top-level switch overrides a
  nested switch. If the nested value survives, Qwen3.8-Flash-Next requires
  a boolean or returns a `400`. With no recorded facts its type is preserved.
- The server resolves the arguments once per request. Tool checks can
  render the conversation more than once. On the native path a recognized
  conversation refusal gives a `422` before any comparison. For other
  failures with request arguments, the server tries the render without them. If
  it succeeds, the server discards that prompt and returns a `400` naming
  the fields and quoting up to 300 characters of the template error. On
  the native vision-language path it reports the thinking-off default if
  it supplied it to the renderer with unrelated template arguments. If
  recorded facts changed the arguments, it shows up to 300 characters of the
  resolved arguments. If the comparison also fails, the server handles its
  error as it would without the fields. A recognized native conversation
  refusal gives a `422`. A `TypeError` on a MiniMax text render with tools
  alone triggers explicit instructions with the request's arguments. For other `TypeError`,
  `ValueError` or Jinja template errors, both native and text paths return a
  `503` with the exception type and the first 300 characters of its message.
  Other exceptions propagate.
- With `minimax-m2` on a text template that takes tools, the server first
  renders an unforced request with the real tool names. After that succeeds,
  it renders the same conversation and arguments again with a unique
  identifier for each tool name. The server uses the real prompt only if
  the second render succeeds, contains every identifier and exactly reproduces
  the real prompt when the original names are put back.
  In the renamed render each name must be the name of a JSON declaration
  inside its own `<tool>...</tool>` block.
  Otherwise it renders explicit instructions with the real tools and the
  request's arguments. The parser reads only MiniMax's call format, so a
  template that presents tools another way (Qwen3-Coder's, for example)
  gets explicit instructions. A forced `tool_choice` always uses explicit
  instructions listing every supplied tool in request order, then directing
  the model to call the selected function. MiniMax also uses explicit
  instructions on vision launches and text launches without template tools.
  `openai-json` always uses instructions.
- Explicit instructions list every supplied tool in request order,
  even when `tool_choice` selects one function. Changing this list between
  turns can shorten the reusable prefix or leave no usable vision checkpoint.
  When tools are enabled and the request supplies a nonempty `tools` list,
  the selected tool must be in that list. `openai-json` accepts only
  functions and lists them before the forced instruction. The server returns
  a `400` if it cannot prepare the instructions, if they omit a supplied
  tool or if their text is missing from the final prompt (differences in
  whitespace are accepted). Set `tool_choice`
  to `none` or launch with `--tool-call-parser none` to turn tools off.
- `chat_template_kwargs` names must be nonempty. The launcher reserves
  names for renderer parameters, including `tools`, `messages`, `tokenize`,
  `add_generation_prompt` and `chat_template`. It also reserves the named
  parameters found in the renderer signature it inspects, except that
  `enable_thinking` and `reasoning_effort` are always allowed as names.
  Setting a reserved name gives a `400`.

### On oMLX

On oMLX the fields are oMLX's own. The launcher passes the request through
unchanged, so what they do is upstream's behavior, and it is not the MLX
runtime's (oMLX 0.7.3, Qwen3.8-Flash-Next):

| The request sends | MLX runtime | oMLX |
|---|---|---|
| `reasoning_effort` at a native level, alone | reasoning on at that level | the effort is a template argument and nothing more: it does not turn reasoning on. With the shipped profile's default of thinking off, nothing changes (measured: 0 reasoning tokens for `low`, `medium` and `xhigh` sent alone). Send `"enable_thinking": true` with the effort |
| `reasoning_effort` `none` | reasoning off alone or with `enable_thinking` false, `400` with true | not an off switch: reasoning stays as `enable_thinking` or the default has it, and with reasoning on oMLX retries `none` as `low` (read in oMLX's source, not measured by us). Send `"enable_thinking": false` |
| an effort other than a native level or `none` | `minimal` is sent as `low`, `high` as `xhigh`. Other values give a `400` | retried under oMLX's own nearest name and, if the template refuses that too, rendered without the effort, with no error (read in oMLX's source, not measured by us) |
| the same name as a field and inside `chat_template_kwargs` | a non-null top-level field wins | for `reasoning_effort`, the one inside `chat_template_kwargs` wins (read in oMLX's source) |

For the same reasoning setting on both runtimes, send `enable_thinking`
true with a common native level (`low`, `medium` or `xhigh`), or send
`enable_thinking` false alone for off. Keep top-level and nested fields
consistent.

Measured on Qwen3.8-Flash-Next (E5 build, one 150-word writing prompt,
temperature 0):

| Request | MLX runtime with the MTP drafter | oMLX |
|---|---|---|
| thinking off (the default) | 0 reasoning tokens, 4 s | 0, 5 s |
| `enable_thinking` true, `reasoning_effort` `low` | 2,250 reasoning tokens, 37 s | 1,059, 35 s |
| the same, `medium` | 2,809, 44 s | 2,218, 66 s |
| the same, `xhigh` (also what thinking on without an effort gives) | 1,821, 28 s | 1,395, 42 s |
| `reasoning_effort` `low`, `medium` or `xhigh` alone | the same prompt as with `enable_thinking` true, so the three rows above | 0: reasoning stays off |

Each level gave a different answer on this prompt. On both runtimes
`xhigh` used fewer reasoning tokens than `medium`. The effort sets no
separate reasoning budget. A positive `max_tokens` caps the whole answer.

## Qwen3.8-Flash-Next with a drafter

Qwen3.8-Flash-Next has a multi-token-prediction (MTP) head. In the tests
below, its drafter sped up generation and matched plain decoding at
temperature 0. Set a positive `max_tokens` when comparing outputs.
Which runtime can use the head depends on where it is:

| The head is | Runtime | How |
|---|---|---|
| a separate drafter directory (`python -m mlx_vlm.split_mtp`, model type `qwen4_exp_mtp`), the usual case for MLX conversions, which strip it from the checkpoint | MLX | `--draft-model <drafter directory>`. The command below selects the MLX runtime and prints `runtime=mlx (auto: model_type=qwen4_exp, --draft-model given, mlx only)` |
| embedded in the checkpoint | oMLX | `KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"mtp_enabled": true}'`. Refused if checkpoint metadata lists no matching tensor name (see "What a launch on oMLX does"). Not measured by us: no such checkpoint was at hand |

The command for the first case, with the n-gram table kept on disk:

```bash
kamiwaza-mlx-server -m ./Qwen3.8-Flash-Next-PLEq8-E5-Dbf16 \
  --draft-model ./Qwen3.8-Flash-Next-PLEq8-E5-Dbf16-mtp \
  --ple-store ./e5-table.json
```

Measured through the server on a Mac Studio (M3 Ultra, 512 GB), E5 build,
table on disk, temperature 0, generation rate as the server logs it:

| Configuration | Prose | Code | Accepted drafts (prose / code) | Memory |
|---|---|---|---|---|
| oMLX, no drafter | 38 tok/s | 38 tok/s | | 91 GB |
| MLX runtime, no drafter | 33 tok/s | 33 tok/s | | 90 GB |
| MLX runtime, MTP drafter, default block (4) | 53 tok/s | 73 tok/s | 55% / 88% | 92 GB |
| the same, `--draft-block-size 3` | 51 tok/s | 64 tok/s | 64% / 92% | 92 GB |
| the same, `--draft-block-size 5` | 52 tok/s | 78 tok/s | 46% / 82% | 92 GB |
| the same, `--draft-block-size 8` | 39 tok/s | 74 tok/s | 27% / 65% | 92 GB |

- The output was the same text with and without the drafter, and at every
  block size, on all four test prompts (prose, code, and two tool-call turns
  with 25 tools).
- The table stays on disk with the drafter on (the memory column).
- Block size. The drafter's `config.json` says `block_size: 2`: that is the
  depth the head was trained at (one extra token). At run time mlx-vlm applies
  the head repeatedly, up to a block of 4 for this drafter when
  `--draft-block-size` is not given. Set at least 2 to allow one target token
  and at least one draft token per round. The block is a ceiling, not a fixed
  size: mlx-vlm drafts at the trained depth for the first rounds of a
  generation and goes up to the block only while at least 65% of recent
  rounds had their first draft accepted. 5 was a little faster on code and no
  faster on prose; 8 lost on prose, where most of a long draft is rejected.
  The default is a reasonable choice for mixed work.
- Before the MLX runtime supported vision prompt reuse, a Mac Studio run on
  commit `a05a122` with the Flash-Next E5 build took about 13–15 s per
  short-answer tool-calling turn with the MTP drafter. MLX kept the weights
  loaded and evaluated the full prompt (about 13,000 tokens, 25 tools) on
  each turn. On warm short-answer turns, oMLX without a drafter kept the
  prefix and answered in about 1–5 s. Long answers took longer on both runtimes.
- On development commit `9174b27`, the same Mac Studio and Flash-Next E5
  build (n-gram table on disk) ran a scripted tool-calling conversation of
  ten steps with a long system prompt and 25 tools. The prompt grew from
  13,691 to 18,143 tokens. At each step we timed two complete responses at
  temperature 0 without streaming. The second request repeated the first
  request's prompt, so with reuse on it could restore a checkpoint the first
  request had just stored. At steps 3 and 7 the prompt crossed a boundary.
  With reuse on, the second request restored 2,048 more tokens and finished
  sooner than the one-token request at both steps. The first two rows give
  the minimum / median / maximum for steps 2 to 10. We measured process
  footprint at the end of each configuration, after the scripted conversation
  for reuse off and after the additional tool-call phases for reuse on.

  | | Reuse off, no drafter | Reuse on, no drafter | Reuse off, MTP drafter | Reuse on, MTP drafter |
  |---|---|---|---|---|
  | One-token request (taken as time to first token) | 14.1 / 15.8 / 17.9 s | 1.06 / 1.95 / 2.74 s | 13.7 / 15.6 / 17.6 s | 1.03 / 1.92 / 2.71 s |
  | The same prompt again, up to 400 tokens | 14.4 / 16.4 / 18.4 s | 1.17 / 2.30 / 2.67 s | 14.2 / 16.1 / 18.1 s | 1.00 / 1.93 / 2.43 s |
  | First prompt of the conversation, one token | 62.0 s, including the first read of the weights | 16.2 s | 13.6 s | 13.7 s |
  | Prompt tokens restored | none | 12,288, then 14,336, then 16,384 | none | 12,288, then 14,336, then 16,384 |
  | Final process footprint | 91 GB | 91 GB | 93 GB | 93 GB |

  The assistant messages matched with reuse on and off at all ten steps for
  both requests, without the drafter and with it (40 comparisons). They also
  matched with and without the drafter. Both comparisons left out the tool-call
  IDs the server generates per response. The first request in each configuration
  started with no checkpoint and read the full prompt. With reuse on, each later
  request of this conversation read on from the last kept 2,048-token
  boundary. The two checkpoints kept took about 0.5 GB each. Reuse depends
  on matching prompt tokens.
  A new conversation can share a kept prefix, and an edit to the history
  can leave no matching checkpoint.
  Commit `9174b27` also returned a tool call written after prose as a call.
  Every answer compared here was plain text without a tool call, and the
  prompts were scripted, with no answer fed back, so that change did not
  shape the messages compared. The times were taken with that commit's
  parser in the response path.
- On commit `a05a122`, oMLX returned valid calls at its server defaults on
  all 60 repetitions of the short two-tool weather request, 30 without
  streaming and 30 with streaming. At the launcher's old MLX defaults
  (`temperature=1.0`, `top_p=1.0`, no top-k), MLX made 59 valid calls
  without a drafter and 54 with the MTP drafter. Sending `"top_p": 0.95`
  brought the drafted run to 60 of 60.
  The code test sent ten prompts twice each, with 25 tools and about 13,000
  input tokens per request. oMLX returned 20 valid calls.
  MLX returned 19 without a drafter and 18 with the MTP drafter at the
  launcher's old defaults. The MTP run with `top_p=0.95` also returned 18.
  All 25-tool misses came from one request sent twice per run. On each miss
  the model answered with a sentence followed by the call. These requests
  used automatic tool choice, so the server returned that answer as text.
  In the separate temperature-zero probes, every tool-call turn tested on
  oMLX without a drafter and on MLX without a drafter or with the MTP
  drafter returned a valid call.
- On the MLX runtime, ordinary streaming sends reasoning in `content`.
  A generation limit can end the answer before `</think>`. The
  `--strip-thinking` option or `strip_thinking: true` can filter paired
  tags and their contents. With auto-detection on, one-piece answers,
  text streams and vision-language streams start inside a think block if
  the request's rendered prompt ends with an open `<think>` tag, and
  outside one if it ends with `</think>`. Think tags earlier in the prompt
  are not read. Some renderers return the last message itself as the
  prompt (mlx-vlm's helper does for PaliGemma, Florence-2 and Falcon OCR);
  when the prompt ends with the text of the request's last message (where the message keeps its parts, its last text part that is not blank) and that text
  ends with the tag after text of its own, the tag is treated as the
  client's text and is not read either (a heuristic: the renderer does not
  say who wrote the end of the prompt). A prompt that ends with neither uses the
  process-wide detection result, which includes the start-up probe of the
  template rendered without request arguments. When a block was inferred
  from the prompt and the output has no think tag at all, an output of 50
  tokens or fewer is withheld and a longer one is returned as the answer. On
  a stream without tools, a block whose opening tag the filter recognizes
  and that does not close is not sent in `content`. A one-piece answer
  (non-streamed, or streamed with tools) is split by the same rules as
  before: when its first think tag is `</think>`, the text after that tag,
  without leading newlines, is the content, think tags included; otherwise
  paired blocks are removed, and an unmatched tag and the text after it can
  stay in `content`.
  The DFlash stream path of this rule is covered by unit tests only; it was not run with a live model.
  With tools enabled the runtime generates the answer in one piece and
  returns any extracted reasoning as `reasoning_content`. oMLX sends
  reasoning as `reasoning_content`.
- A DFlash 2 drafter for this model (`incoai/Qwen3.8-Flash-Next-DFlash2`,
  revision `a22a1744`) loads with mlx-vlm 0.7.4, but drafting fails on the
  model's hidden-state layout. The start-up self-test requires a completed
  draft round and exits 2 with mlx-vlm's error when generation raises.
  oMLX's DFlash does not cover this model type either.

## Native speculative decoding (MTP)

Vision-language checkpoints loaded through `mlx-vlm` can use mlx-vlm's own
speculative decoding with a separate drafter: a model's native MTP
(multi-token prediction) head, a DFlash2 drafter, or an EAGLE-3 drafter.
In greedy tests with the same positive token limit, Qwen3.8-Flash-Next
produced the same output with and without its MTP drafter.
Many MLX conversions strip the MTP head, so extract it from a checkpoint
that still carries it:

```bash
python -m mlx_vlm.split_mtp --model ./Qwen3.8-Flash-Next-FP8 --output ./Qwen3.8-Flash-Next-MTP

python -m kamiwaza_mlx.server \
  -m ./Qwen3.8-Flash-Next-mlx \
  --draft-model ./Qwen3.8-Flash-Next-MTP
```

| Flag | Env var | Default | Meaning |
|------|---------|---------|---------|
| `--draft-model` | `KAMIWAZA_MLX_DRAFT_MODEL` | off | Drafter path or HF id. |
| `--draft-kind` | `KAMIWAZA_MLX_DRAFT_KIND` | `auto` | `auto`, `mtp`, `dflash` or `eagle3`; `auto` reads the drafter's config. |
| `--draft-block-size` | `KAMIWAZA_MLX_DRAFT_BLOCK_SIZE` | mlx-vlm's default for the drafter (4 for the Qwen3.8-Flash-Next MTP head) | Maximum tokens per round, including one target token. The minimum is 2. With 1, mlx-vlm drafts nothing and ends every answer after its first token. The Qwen3.8-Flash-Next MTP head uses shorter rounds when too few recent rounds accept the first draft token. |
| `--draft-queue-timeout` | `KAMIWAZA_MLX_DRAFT_QUEUE_TIMEOUT` | `600` | Seconds a request waits for the drafter before failing with `503` and `Retry-After: 5`; `0` waits indefinitely. The wait ends when the request takes the drafter. Parsing the request and checking its token limit and reasoning fields come before the wait; loading its images and building its prompt come after. |
| `--draft-stream-send-timeout` | `KAMIWAZA_MLX_DRAFT_STREAM_SEND_TIMEOUT` | `60` | Seconds a draft-enabled stream may wait for the client to accept one chunk before it is aborted and the drafter released; `0` disables. |

- With a native drafter, an unset, null, zero or negative token limit becomes
  the served context window minus the token count of the prompt text, with a
  minimum of 1. Image tokens are excluded, so an image request can get more
  room than the window has left. If the window is unknown, the limit is
  effectively unbounded (`2**31 - 1`). A positive limit and every limit on
  a launch without a native drafter go to generation as before. With a
  native drafter the server treats zero as unset. On a vision-language launch
  without one it passes zero through to mlx-vlm; for an autoregressive
  model its generation then emits no tokens. On the `kamiwaza-mlx-server` chat endpoint
  on the MLX runtime, `max_tokens` and `max_completion_tokens` name this
  same limit. Send either or both when equal. Null for either means unset.
  Different integers give a `400`, for example
  `max_tokens=30 and max_completion_tokens=50 differ. Send one of the two fields.`
  The vision path does not report `finish_reason: length`, so a text answer
  that reaches the substituted limit reports `stop`.
- Startup fails if the drafter cannot pair with the target model, if the model
  loads through `mlx-lm` (for example with `--text-only`), or if
  `--dflash-draft-model` is also set. The server starts serving only after
  the requested drafter completes a round in its self-test.
- The drafter serves one generation at a time, so draft-enabled requests
  queue. A streamed response holds the drafter until its last chunk is sent,
  so a slow reader delays everyone behind it (each chunk must be accepted
  within `--draft-stream-send-timeout`, or the stream is aborted). `--draft-queue-timeout` bounds
  the wait with a `503`, but the timer runs on the server's event loop: while a
  non-streamed or tool-call generation is running it cannot fire, so a waiter
  may be answered (or rejected) only after that generation finishes. A client
  that disconnects while queued is skipped when its turn comes, on a best-effort
  basis (a disconnect the server has not yet read is not detected). A request
  that completes a draft round logs the mean tokens per round (one target
  token plus the accepted drafts) and the fraction of drafted tokens accepted.
- Startup also fails if the hidden size or vocabulary size the launcher
  reads from the drafter's configuration (its text configuration first,
  then the top level) differs from the target model's (for example a
  drafter built for another model family; a drafter that records the
  target's width as `backbone_hidden_size` or `target_hidden_size` is
  compared by that value; sizes recorded elsewhere, such as an EAGLE-3
  drafter's vocabularies, are not compared),
  or if an explicit `--draft-kind` disagrees with the drafter's own type.
- After loading the drafter, the server tries up to three greedy text
  generations of up to eight tokens each. A generation passes when it
  finishes with at least one completed draft round, even if the target
  rejected every proposal.
  If generation raises or all three finish without a round, the launch
  exits 2 with the reason. The test warms the weights its prompts use.
  The log line starting with `Drafter self-test passed in` reports the
  elapsed time and completed rounds. The earlier single-generation test took
  3 to 13 s for Qwen3.8-Flash-Next in our runs, depending on the system's
  file cache.
  Sampled decoding, image inputs and the HTTP layer are outside this test.
- Successful non-streamed responses for requests that complete a draft round include
  `X-Speculative-Tokens-Per-Round`, `X-Speculative-Acceptance-Rate` and
  `X-Speculative-Rounds`. A request limited to one token has none of these
  headers.
- Measured through the server on Qwen3.8-Flash-Next with its MTP head (M5 Max,
  128 GB, temperature 0): 37 tok/s without a drafter; 52 tok/s on prose and 73
  tok/s on code with it. Gains are smaller with sampling (temperature 1.0: 36
  and 62 tok/s).

## N-gram table on disk (Qwen3.8-Flash-Next)

Qwen3.8-Flash-Next (mlx-vlm model type `qwen4_exp`) carries a very large
per-layer n-gram embedding table, also called the PLE or "engram" table, of
about 51 billion values. It is a lookup table. Loaded as ordinary
parameters it occupies tens of gigabytes of unified memory; `mlx-vlm` can
instead memory-map the table inside the checkpoint's safetensors files and read
only the rows each token needs.

A model directory opts in through `text_config.ple_storage` in its
`config.json` (mlx-vlm's `prepare_external_ple_model` writes such a directory).
`--ple-store` makes the same choice at launch, without preparing a second
directory, and can point at the table of another conversion of the same model:

```bash
# Index the table of a converted checkpoint in place (writes a small JSON file;
# no weights are copied). The manifest refers to the checkpoint by relative
# path, so keep the two where they are.
python -m kamiwaza_mlx.ple_store ./Qwen3.8-Flash-Next-mlx-8bit ./tables/q8/ple-store.json

# Serve 4-bit weights with that 8-bit table read from disk.
python -m kamiwaza_mlx.server \
  -m ./Qwen3.8-Flash-Next-mlx-4bit \
  --ple-store ./tables/q8/ple-store.json
```

| Flag | Env var | Default | Meaning |
|------|---------|---------|---------|
| `--ple-store` | `KAMIWAZA_MLX_PLE_STORE` | off | Manifest of the n-gram table to read from disk. Replaces the table in the model directory, and a manifest the model's `config.json` declares. |

- The table bundled in the model directory is not loaded when `--ple-store` is
  set. To keep a model's own table on disk, index that same directory and pass
  the manifest.
- Use a table from a conversion of the same model. The launcher can only
  check its shape: a table with a different number of rows or row width is
  refused, but a table of the same shape from another model is not detected
  and would give wrong output.
  Supported table layouts are 4-bit and 8-bit affine with group size 32, and
  4-bit `nvfp4`. Anything else fails at startup with the reason.
- Startup fails, and the server never serves without the requested table, if
  the manifest is missing or incomplete, if the model is not a `qwen4_exp`
  model, if `--text-only` or `--dflash-draft-model` is also set, if the table
  does not fit the model, or if the installed `mlx-vlm` has no on-disk table
  support (the pinned 0.7.4 has it). A table that does not fit is reported as
  `the n-gram table of --ple-store <manifest> could not be used: <reason>`; a
  failure of the model itself keeps its own message.
- The model must be one this server loads through `mlx-vlm`, which is what it
  does for a conversion that keeps its vision tower. For any other model
  startup fails; it is not forced through `mlx-vlm`.
- Lookups read from the page cache, so the first tokens after a cold start are
  slower. `--cache-rows N` on the manifest command records a row cache size in
  the manifest: the N most recently used rows are kept in memory. With
  `--ple-store` the manifest's value is the one used, whatever the model
  directory's own `config.json` declares.

## Experimental DFlash speculative decoding

DFlash is available as an opt-in text-only generation path. Configure a draft
model and the server will use a DFlash speculative loop instead of the normal
`mlx_lm.generate.stream_generate` loop. The default backend is `auto`: it uses
the standalone `dflash-mlx` implementation when installed, otherwise it falls
back to ZLab's `dflash.model_mlx`.

With `--strip-thinking` or `strip_thinking: true`, the stream of this path
decides per request whether an answer starts inside a think block (see
`--strip-thinking` under "Most useful flags"). That rule is covered by unit
tests only on this path; it was not run with a live model.

The DFlash backends install from git (PyPI does not allow direct-URL
dependencies in package metadata). These are the tested pins:

```bash
# ZLab backend
pip install "dflash[mlx] @ git+https://github.com/z-lab/dflash.git@6d6229eaddce58fcd7e4cc91945da632f5544c86"

# Standalone dflash-mlx backend
pip install "dflash-mlx @ git+https://github.com/bstnxbt/dflash-mlx.git@fada1eb2b75cd1c875ca6547b6518783fd3d2956"

python -m kamiwaza_mlx.server \
  -m mlx-community/Qwen3.5-27B-5bit \
  --dflash-draft-model z-lab/Qwen3.5-27B-DFlash \
  --dflash-backend auto \
  --dflash-block-size 16 \
  --dflash-default-max-tokens 2048 \
  --dflash-max-context -1
```

The first five environment variables correspond to the DFlash options above.
The last three explicitly set verification and stream synchronization to their defaults.

```bash
export KAMIWAZA_MLX_DFLASH_DRAFT_MODEL=z-lab/Qwen3.5-27B-DFlash
export KAMIWAZA_MLX_DFLASH_BACKEND=auto
export KAMIWAZA_MLX_DFLASH_BLOCK_SIZE=16
export KAMIWAZA_MLX_DFLASH_DEFAULT_MAX_TOKENS=2048
export KAMIWAZA_MLX_DFLASH_MAX_CONTEXT=-1
export KAMIWAZA_MLX_DFLASH_VERIFY_MODE=auto
export KAMIWAZA_MLX_DFLASH_VERIFY_QMM=true
export KAMIWAZA_MLX_DFLASH_STREAM_SYNC_EVERY=0
```

Notes:

- DFlash is only enabled when `--dflash-draft-model` is non-empty, and that
  forces a text-only `mlx-lm` target load even for Qwen3.5 conversions that
  retain a `vision_config`.
- The Qwen3.5-27B drafter is trained for target `Qwen/Qwen3.5-27B`; MLX
  quantized conversions such as `mlx-community/Qwen3.5-27B-5bit` and
  `mlx-community/Qwen3.5-27B-6bit` are structurally compatible targets to try,
  but acceptance and quality need measurement per quant.
- The `z-lab/Qwen3.5-27B-DFlash` draft config uses five `full_attention`
  layers. The `dflash-mlx` runtime has draft sink/window machinery for other
  draft shapes, but those knobs do not change this Qwen3.5 speculative model's
  draft attention.
- Server KV and prefix caches are disabled while DFlash is enabled because
  DFlash allocates its own per-request target/draft prompt caches and mutates
  target model hooks during generation.
- `--dflash-backend auto` picks `dflash-mlx` when the standalone `dflash_mlx`
  package is installed (see the tested git pin above); otherwise it falls back
  to the ZLab backend.
  The inline `dflash-mlx` adapter runs without that package's own
  prefix snapshot cache, so it is meant for local kernel/runtime experiments,
  not as a replacement for that package's full OpenAI-compatible server.
  It also installs a Qwen target-op compatibility patch that skips the
  full-vocabulary lm-head projection for large prompt-prefill chunks whose
  logits are discarded by the DFlash loop, and disables unused prefix-snapshot
  work because the inline adapter does not provide a `dflash-mlx` prefix cache.
- For eligible 4-bit or 8-bit MLX target linears, the `dflash-mlx` adapter also
  installs that package's verifier-linear wrappers. Use
  `--dflash-verify-mode off` to disable the swap, or
  `--no-dflash-verify-qmm` to keep the wrappers while disabling the custom qmm
  kernels.
- `--dflash-max-context` is a degradation guard: prompts at or above that token
  count bypass DFlash and use the normal MLX generation path. Omit it, or set
  it to `-1`, to use the conservative auto cap of `4000` prompt tokens. Set
  it to `0` only when you intentionally want to route every prompt through
  DFlash. Because DFlash mode disables the server-level shared KV/prefix
  caches, bypassed requests do not use shared prefix-cache optimizations, but
  they still allocate the normal bounded per-request generation cache.
- DFlash generation is serialized with an internal lock. That protects the
  target model hidden-state hooks and Qwen3.5 gated-delta rollback patch.
- Extra `mx.synchronize()` calls in the DFlash streaming loop are disabled by
  default. Set `--dflash-stream-sync-every N` only when diagnosing stream timing
  or device scheduling.

**Note:** Not specific to Kamiwaza (that is, you can use on any Mac, Kamiwaza not required)
```bash
pip install kamiwaza-mlx

# start the server
a) python -m kamiwaza_mlx.server -m ./path/to/model --port 18000
# or, if you enabled the optional entry-points during install
b) kamiwaza-mlx-server -m ./path/to/model --port 18000

# chat from another terminal (note: specify --host to match server port)
python -m kamiwaza_mlx.infer --host localhost:18000 -p "Say hello"
```

The remainder of this README documents the original features in more detail.

# MLX-LM 🦙 — Drop-in OpenAI-style API for any local MLX model

A FastAPI micro-server (server.py) that speaks the OpenAI
`/v1/chat/completions` dialect, plus a tiny CLI client
(`infer.py`) for quick experiments.
Ideal for poking at huge models like Dracarys-72B on an
M4-Max/Studio, hacking on prompts, or piping the output straight into
other tools that already understand the OpenAI schema.

---

## ✨ Highlight reel

| Feature | Details |
|---------|---------|
| 🔌 OpenAI compatible | Same request / response JSON (streaming too) – just change the base-URL. |
| 📦 Zero-config | Point at a local folder or HuggingFace repo (`-m /path/to/model`). |
| 🖼️ Vision-ready | Accepts `{"type":"image_url", …}` parts & base64 URLs – works with Qwen-VL & friends. |
| 🎥 Video-aware | Auto-extracts N key-frames with ffmpeg and feeds them as images. |
| 🧮 Usage metrics | Prompt / completion tokens + tokens-per-second in every response. |
| ⚙️ CLI playground | `infer.py` gives you a REPL with reset (Ctrl-N), verbose mode, max-token flag… |

---

## 🚀 Running the server

```bash
# minimal
python -m kamiwaza_mlx.server -m /var/tmp/models/mlx-community/Dracarys2-72B-Instruct-4bit

# custom port / host
python -m kamiwaza_mlx.server -m ./Qwen2.5-VL-72B-Instruct-6bit --host 0.0.0.0 --port 12345
```
Default host/port: `0.0.0.0:18000`

### Most useful flags:

| Flag | Default | What it does |
|------|---------|--------------|
| `-m / --model` | `mlx-community/Qwen2-VL-2B-Instruct-4bit` | Path or HF repo. |
| `--host` | `0.0.0.0` | Network interface to bind to. |
| `--port` | `18000` | TCP port to listen on. |
| `-V / --vision` | off | Force vision pipeline; otherwise detect declared vision config. |
| `--text-only` | off | Explicit language-only load for converted checkpoints retaining vision metadata; mutually exclusive with `--vision`. |
| `--strip-thinking` | off | Removes `<think>…</think>` blocks from model output. A request can set `strip_thinking: true`. With auto-detection on, one-piece answers, text streams and vision-language streams start inside a think block if the request's rendered prompt ends with an open `<think>` tag, and outside one if it ends with `</think>`. Think tags earlier in the prompt are not read, nor a tag that ends the text of the request's last message (where the message keeps its parts, its last text part that is not blank) where the prompt ends with that text and the text has more of its own before the tag (a heuristic for renderers that return the last message as the prompt). A prompt that ends with neither uses the process-wide detection result, which includes the start-up probe of the template rendered without request arguments. The DFlash stream path of this rule is covered by unit tests only; it was not run with a live model. |
| `--draft-model` | off | Native mlx-vlm speculative drafter (MTP, DFlash2, EAGLE-3); see [Native speculative decoding](#native-speculative-decoding-mtp). |
| `--vlm-prompt-reuse` | `auto` | Reuse matching prompt prefixes on eligible MLX vision-language models for requests without images or media placeholders. See [Prompt reuse for vision-language models](#prompt-reuse-for-vision-language-models). |
| `--sampling-defaults` | `model` | Choose defaults for sampling fields omitted from a request. See [Sampling defaults](#sampling-defaults). |
| `--enable-prefix-caching` | `True` | Enable automatic prompt caching for text-only models. If enabled, the server attempts to load a cache from a model-specific file in `--prompt-cache-dir`. If not found, it creates one from the first processed prompt and saves it. |
| `--prompt-cache-dir` | `./.cache/mlx_prompt_caches/` | Directory to store/load automatic prompt cache files. Cache filenames are derived from the model name. |

---

### KV cache flags (all KV-related CLI knobs)

**System-prefix cache (system-only)**

| Flag | Default | What it does |
|------|---------|--------------|
| `--enable-prefix-caching` | `True` | Enable system-prefix caching for text-only models. |
| `--prompt-cache-dir` | `./.cache/mlx_prompt_caches/` | Directory for system-prefix cache files (`.safetensors`, `.len`, `.hash`). |
| `--system-cache-max-tokens` | `2048` | Max tokens to cache from the system prompt (`0` = unlimited). If the system prompt exceeds this cap, system caching is skipped. |
| `--prefix-cache-headroom` | `64` | Extra tokens reserved beyond the system prompt length when sizing the system cache. |

**Conversation cache (simple global prefix-match)**

Simple mode uses a single text-model conversation cache. If the previous prompt is a prefix of the incoming prompt, the server skips prefill for that prefix. Otherwise it resets the cache. Conversation IDs appear in logs and metadata.

| Flag | Default | What it does |
|------|---------|--------------|
| `--disable-kv-cache` | off | Disable the launcher's retained KV caches, including vision prompt checkpoints. Generation still uses temporary KV state within each request. |
| `--kv-cache-max-tokens` | `0` | Per-cache upper bound (`0` = min(model context, 128k)). |
| `--kv-cache-keep` | `4` | Tokens to keep when trimming rotating caches. |
| `--kv-cache-idle-release-seconds` | `0` | Release KV caches, including retained vision prompt checkpoints, on the next chat request after the idle interval (`0` disables idle release). |
| `--kv-cache-hard-reserve` | `True` | Fully reserve KV memory up to sizing target. |
| `--kv-cache-warmup` | `False` | Run a warm-up pass at startup to materialize KV shapes. |
| `--kv-cache-warmup-tokens` | `0` | Warmup tokens (`0` = use resolved target size). |
| `--retain-mx-cache` | `False` | Keep MX allocator memory (disables `mx.clear_cache`). |

### Prompt reuse for vision-language models

mlx-vlm can save the target model's prompt state at a 2,048-token boundary.
An eligible request can continue from the longest retained prefix that matches
token for token. Prompts of 2,048 tokens or fewer are always read in full.
For 2,049 tokens, the end-minus-one checkpoint falls exactly at 2,048 and
can be kept. The alignment check rejects an end-minus-one checkpoint only
when it falls off the grid. Even an identical prompt leaves its final token
to read.

Requests with images or a media placeholder such as `<|image_pad|>` anywhere
in the rendered prompt bypass reuse. Either can cause mlx-vlm 0.7.4 to split
a prefill chunk at a different token, even on a cache miss. When the input
IDs contain no media placeholders, chunked prefill uses mlx-vlm's default
step, and the request uses MTP or no drafter, restoring a grid boundary
preserves the forward calls for the remaining prompt.
The offline Metal test compares generated tokens with
and without reuse at temperature 0 (greedy decoding) on tiny random Qwen
weights for repeated prompts, growing conversations and shared-prefix forks.

Each running request that restores a checkpoint uses its own copy. Reuse adds
no new request-wide lock. The manager takes its own lock around storing and
looking up checkpoints.

| Flag | Environment variable | Default | Meaning |
|---|---|---|---|
| `--vlm-prompt-reuse {auto,on,off}` | `KAMIWAZA_MLX_VLM_PROMPT_REUSE` | `auto` | Enable reuse when the startup checks pass. Use `on` to require the manager at startup or `off` to disable it. |
| `--vlm-prompt-reuse-checkpoints N` | `KAMIWAZA_MLX_VLM_PROMPT_REUSE_CHECKPOINTS` | `2` | Keep at most N checkpoints in memory across requests. The minimum is 2 because mlx-vlm reserves one slot when choosing up to N−1 intermediate boundaries per request. |
| `--vlm-prompt-reuse-max-gb G` | `KAMIWAZA_MLX_VLM_PROMPT_REUSE_MAX_GB` | `8` | Maximum total size of retained checkpoint cache buffers in GiB. Token lists and bookkeeping are outside this limit. The value converted to bytes must be finite and at least one byte. |

Blank or whitespace-only environment values use the defaults above. The MLX
runtime validates both numeric settings at startup in every mode, including
`off`. Reuse requires KV caching and a listed vision-language model type.
The list currently contains only `qwen4_exp` (Qwen3.8-Flash-Next). See
[`config/prompt_reuse.json`](kamiwaza_mlx/config/prompt_reuse.json).

The eligibility check runs after the model and any vision drafter have loaded
and the drafter's self-test has passed. It accepts MTP or no drafter. DFlash
and EAGLE-3 need hidden states for the entire prompt. The launcher verifies
that the tokenizer resolves each configured media token ID to a placeholder
string and that tokenizing a plain, media-free probe produces none of those
IDs. It builds the manager, checks the whole-model checkpoint plan and the
model's cache layout, and exercises the manager's statistics, byte accounting
and clear operations. If reuse is unavailable, `auto` leaves it off and
`on` exits 2. Starting successfully with `on` guarantees that these
checks passed. A checkpoint can still be too large to retain. A cap of one
byte stores no checkpoints.

Conversation IDs and `reset_conversation` apply to the text-model
conversation cache. The first vision request in a new conversation can restore
a kept prefix if its tokens match, for example when it starts with the same
system prompt and tools. This also applies with `reset_conversation: true`.
If an edit to the history leaves no matching checkpoint, the next request
reads the full prompt. To force the vision model to read every prompt in
full, start the MLX runtime with `--vlm-prompt-reuse off`.

The launcher decides whether to create the manager and keeps it in memory.
It constructs the manager directly, bypassing the mlx-vlm factory that
reads `APC_ENABLED`, `APC_DISK_ENABLED`, `APC_DISK_PATH`, `APC_BLOCK_SIZE`
and `APC_NUM_BLOCKS`. These variables have no effect on vision prompt reuse.
The launcher supplies the settings for `APC_CHECKPOINT_ENTRIES` and `APC_MEMORY_MAX_GB`
from the two numeric options. It sets the guard (`APC_CHECKPOINT_GUARD_TOKENS`) to
1 token, the interval (`APC_CHECKPOINT_INTERVAL_TOKENS`) to the default
prefill step and the reserve (`APC_MEMORY_RESERVE_GB`) to mlx-vlm's automatic
value. The manager reads some `APC_*` settings directly. For example, the
launcher caps a valid `APC_EXACT_MIN_TOKENS` at one prefill step. A
nonnumeric value makes the startup check fail. The launcher reports mlx-vlm's
error text and appends the names of inherited `APC_*` variables. The error
text itself can include a value.

Retained checkpoints are evicted least recently used first. Under memory
pressure mlx-vlm can also evict a checkpoint or skip a store or restore.
Even for a single Qwen request, mlx-vlm makes room before extracting a
checkpoint from the prompt cache. Under pressure it can evict a matching
prefix at that step, even if the launcher then declines to store an off-grid
end-of-prompt checkpoint. If all matching prefixes were evicted, the next
replay reads the whole prompt.

A successful non-streamed vision response reports the restored prefix length
in `usage.prompt_tokens_details.cached_tokens`, including zero on a miss or
bypass. For a streamed request, the per-request restore count is available
only in the generation log. The `X-Conv-*` headers describe the text
conversation cache. With reuse enabled, the launcher logs the restored and
total prompt tokens for each eligible generation that produced a result. The
log records `outcome=completed`, `outcome=abandoned` or `outcome=failed`,
even if the generation ended before its first result. The launcher logs a
bypass reason for images or media placeholders. A warning records an increase
in the manager's memory skip or memory eviction counter only when the snapshots
before and after the request both report a nonnegative integer for that counter
(booleans are invalid).

`GET /v1/kv_cache/stats` reports the startup decision and limits in
`vlm_prompt_reuse`. `checkpoints` is the configured entry limit.
`resident_bytes` is the total size of the kept checkpoints' cache buffers.
It excludes copies in use by running requests and is `null` when reuse is
off. If the statistics read fails or a required field is missing or is
not a nonnegative integer (booleans are invalid), the block includes
`statistics_error`, and `resident_bytes` and
`mlx_vlm_checkpoint_counters` are both `null`.

- `launcher_counters` counts eligible requests and tracks image and media
  placeholder bypasses separately. `restored_requests` counts eligible
  requests whose generation result reported a restored prefix. If mlx-vlm
  restores a prefix and generation fails before its first result, the launcher
  counts the request in `eligible_requests` and `eligible_without_result`.
  Only mlx-vlm records the restore in that case, so its counters can differ
  from the launcher's. The launcher's prompt-token and restored-token totals
  cover eligible requests that produced a result.
  The launcher keeps its counts for the lifetime of the process.
- `mlx_vlm_checkpoint_counters` reports the manager's checkpoint stores
  and hits, restored tokens, memory skips, memory evictions, evictions and
  rejects. Ordinary replacement at the entry limit does not increment
  mlx-vlm's eviction counter. The manager's counters restart at zero when
  it clears the checkpoints, including on idle release. If the idle clear
  raises an exception, the launcher warns that checkpoints may have been
  kept and continues serving the request.

`totals.kv_bytes` includes the resident checkpoint bytes. It is `null`
when `statistics_error` is present, because the checkpoint bytes are then
unknown.

---

### Experimental multi-node via `mlx.distributed`

This section documents the direct `server.py` distributed bootstrap path.

It is distinct from the current Kamiwaza-managed two-node serving flow, which
uses `kamiwaza-mlx-distributed` plus upstream `mlx_lm.server` and is documented
in `MLX_TWO_NODE_README.md`.

The server can bootstrap a two-node mesh using `mlx.distributed`. Set a rendezvous host via `PAIRED_HOST` (optionally in a `.env` file) and launch each node with matching ranks/world-size. The helper will automatically read `.env` files passed via `--distributed-env-file` or located beside the server script.

```bash
# shared settings (either export or place in .env)
PAIRED_HOST=10.0.0.2
PAIRED_PORT=17863
WORLD_SIZE=2

# leader node (rank 0 hosts FastAPI)
RANK=0 python -m kamiwaza_mlx.server --distributed-env-file .env -m ./model

# worker node (rank 1 participates in mlx.distributed but does not bind HTTP)
RANK=1 python -m kamiwaza_mlx.server --distributed-env-file .env -m ./model
```

Useful knobs:

- `--distributed` – force-enable/disable distributed mode (auto when a paired host is set, or `PAIRED_WORLD_SIZE` / `MLX_DISTRIBUTED_WORLD_SIZE` / `--distributed-world-size` is above 1; a bare `WORLD_SIZE` does not turn it on).
- `--distributed-rank` / `--distributed-world-size` – override `RANK`/`WORLD_SIZE` env vars.
- `--distributed-host` / `--distributed-port` – override `PAIRED_HOST` / `PAIRED_PORT`.
- `--distributed-server-rank` – choose which rank should host the HTTP server (defaults to 0).

In this direct path, non-leader ranks simply keep the MLX runtime alive for collective ops once the model weights are synchronized. Treat it as a bootstrap/experimentation path, not as the primary Kamiwaza-managed TP=2 serving mode.

---

## 💬 Talking to it with the CLI

```bash
python kamiwaza_mlx/infer.py --host localhost:18000 --max_new_tokens 2048
```

### Interactive keys
- Ctrl-N: reset conversation
- Ctrl-C: quit

---

## 🌐 HTTP API

GET `/v1/models`

Returns a list with the currently loaded model:

```json
{
  "object": "list",
  "data": [
    {
      "id": "Dracarys2-72B-Instruct-4bit",
      "object": "model",
      "created": 1727389042,
      "owned_by": "kamiwaza"
    }
  ]
}
```
The `created` field is set when the server starts and mirrors the OpenAI API's timestamp.

POST `/v1/chat/completions`

```json
{
  "model": "Dracarys2-72B-Instruct-4bit",
  "messages": [
    { "role": "user",
      "content": [
        { "type": "text", "text": "Describe this image." },
        { "type": "image_url",
          "image_url": { "url": "data:image/jpeg;base64,..." } }
      ]
    }
  ],
  "max_tokens": 512,
  "stream": false
}
```

On the `kamiwaza-mlx-server` chat endpoint on the MLX runtime,
`max_completion_tokens` and `max_tokens` name the same limit. Send either
field, or both with equal values. A null for either field means it is unset.
Different integers return HTTP 400, for example
`max_tokens=30 and max_completion_tokens=50 differ. Send one of the two fields.`

With a native drafter, an unset, null, zero or negative limit becomes the
served context window minus the token count of the prompt text, with a minimum
of 1. Image tokens are excluded, so an image request can get more room than
the window has left. An unknown window gives an effectively unbounded limit
of `2**31 - 1`. A positive limit and every limit on a launch without a native
drafter go to generation as before. With a native drafter the server treats
zero as unset. On a vision-language launch without one it passes zero
through to mlx-vlm; for an autoregressive model its generation then emits no tokens.
The vision path does not report `finish_reason: length`, so a text answer
that reaches the substituted limit reports `stop`.

Response (truncated):

```json
{
  "id": "chatcmpl-d4c5…",
  "object": "chat.completion",
  "created": 1715242800,
  "model": "Dracarys2-72B-Instruct-4bit",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "The image shows…" },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 143,
    "completion_tokens": 87,
    "total_tokens": 230,
    "tokens_per_second": 32.1
  }
}
```

Add `"stream": true` and you'll get Server-Sent Events chunks followed by
`data: [DONE]`.

**System Prefix Caching (Text-Only Models):**
- Purpose: Dramatically speed up repeated queries that share the same system context (e.g., large document in `role: system`). The server caches only the system message(s), not the whole prompt, so subsequent turns process only new user tokens.
- Flags:
  - `--enable-prefix-caching` (default `True`)
  - `--prompt-cache-dir` (default `./.cache/mlx_prompt_caches/`)
  - `--system-cache-max-tokens` (default `2048`, `0` disables the cap)
  - `--prefix-cache-headroom` (default `64`)
- How it works (high‑level):
  1) On first request with a system message, the server builds a KV cache for just the system portion and saves three files under `--prompt-cache-dir`:
     - `<model>.safetensors` (KV), `<model>.safetensors.len` (token count), `<model>.safetensors.hash` (SHA256 over token IDs)
  2) On subsequent requests with the same system text (hash matches), the server deep‑copies the cached KV and processes only new user/assistant tokens.
  3) If the system message changes, the old cache is discarded and replaced automatically.
  4) If the system prompt exceeds `--system-cache-max-tokens`, system caching is skipped and the full prompt is prefetched normally.
- Example: A 10,000‑token system document is processed once; later questions only process the user tokens.
- Notes: text‑only models; fully transparent to clients (no special fields needed).

**Conversation KV Caching (Long chats, fast follow‑ups):**
- Rationale: Reuse KV across turns so only the tail of the prompt is prefetched.
- Behavior:
  - Provide `conversation` or `conversation_id` (or `X-Conversation-Id`) if you want IDs reflected in logs/metadata. If omitted, the server uses `default`.
  - Successful JSON and SSE responses include the text conversation cache headers:
    - `X-Conv-Id` (resolved ID), `X-Conv-KV` (cache mode), `X-Conv-Cached-Tokens`, `X-Conv-Processing-Tokens`.
  - Non‑stream JSON also includes `usage.input_tokens_details.cached_tokens` and `metadata.conversation_id`.
  - The global cache is capped at `min(model context, 128k)` unless overridden by `--kv-cache-max-tokens`.
  - Only one request at a time uses the text conversation cache. Concurrent text requests that bypass a busy cache report `X-Conv-KV: busy`. Vision requests with KV caching enabled report `X-Conv-KV: vision`, including when they restore a prompt checkpoint.
- Breaking change: the legacy `/v1/conv_kv/*` endpoints were removed in this branch.

---

## 🛠️ Internals (two-sentence tour)

* **server.py** – loads the model with mlx-vlm, converts incoming
OpenAI vision messages to the model's chat-template, handles images /
video frames, and streams tokens back. For text-only models, if enabled via server flags, it automatically manages a system message cache to speed up processing when multiple queries reference the same system context.
* **infer.py** – lightweight REPL that keeps conversation context and
shows latency / TPS stats.

That's it – drop it in front of any MLX model and start chatting!
