Metadata-Version: 2.5
Name: mlx-decision
Version: 0.3.0
Summary: Run decision models such as Cloudflare's Clef locally on Apple Silicon with MLX
Project-URL: Homepage, https://github.com/armbues/mlx-decision
Project-URL: Repository, https://github.com/armbues/mlx-decision
Project-URL: Issues, https://github.com/armbues/mlx-decision/issues
Project-URL: Changelog, https://github.com/armbues/mlx-decision/blob/main/CHANGELOG.md
Author: Armin Buescher
License-Expression: MIT AND Apache-2.0
License-File: LICENSE
License-File: LICENSES/clef-Apache-2.0.txt
License-File: LICENSES/mlx-lm-MIT.txt
License-File: LICENSES/transformers-Apache-2.0.txt
Keywords: apple-silicon,classification,clef,decision-model,llm,mlx,qwen
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Environment :: GPU
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: httpx>=0.27
Requires-Dist: huggingface-hub>=1.0
Requires-Dist: mlx>=0.32
Requires-Dist: prompt-toolkit>=3.0.52
Requires-Dist: pydantic>=2.7
Requires-Dist: tokenizers>=0.21
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: fastapi>=0.110; extra == 'dev'
Requires-Dist: numpy>=1.26; extra == 'dev'
Requires-Dist: pillow>=10; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Requires-Dist: uvicorn>=0.29; extra == 'dev'
Provides-Extra: images
Requires-Dist: numpy>=1.26; extra == 'images'
Requires-Dist: pillow>=10; extra == 'images'
Provides-Extra: server
Requires-Dist: fastapi>=0.110; extra == 'server'
Requires-Dist: uvicorn>=0.29; extra == 'server'
Description-Content-Type: text/markdown

# mlx-decision

Run decision models locally on Apple Silicon with [MLX](https://github.com/ml-explore/mlx).

A decision model reads a *state* (a message, a document, any JSON) and a set
of typed *questions*, and returns a probability for every allowed answer in
a single forward pass, without generating text. mlx-decision is to decision
models what `mlx-lm` is to LLMs: load a model, ask it questions from Python
or the command line, serve it over HTTP with an API compatible with
[Jev's](https://typesafe.ai), and convert it to smaller quantized copies.

The first supported model is Cloudflare's
[clef-flash](https://huggingface.co/Cloudflare/clef-flash) (Qwen3.5-9B
backbone plus a joint schema head), with text and image input.

0.3 is the first public release (alpha): the interface may still change.
Questions and bug reports go to the
[issue tracker](https://github.com/armbues/mlx-decision/issues).

## Install

Requirements:
- a Mac with Apple Silicon (M1 or later) and macOS 14 or newer
- Python 3.12 or newer
- memory while answering (smaller copies with [convert](#smaller-models-convert)):
  - clef-flash as released: about 19 GB, so a Mac with 32 GB or more
  - 8-bit copy: 11 GB
  - mixed-precision 4-bit copy: 7 GB

```bash
pip install mlx-decision                    # library and the mlx-decision command
pip install "mlx-decision[images]"          # plus image input (Pillow, NumPy)
pip install "mlx-decision[server]"          # plus the HTTP server (FastAPI, uvicorn)
pip install "mlx-decision[server,images]"   # both
```

From source: clone the [repository](https://github.com/armbues/mlx-decision)
and run `pip install -e ".[server,images]"` in it. MLX also has builds for
Linux and Windows, so pip may install mlx-decision there, but loading a
model stops with an error: the models run on Apple Silicon only.

### Getting a model

Models load from a local folder or a Hugging Face repo id. A repo id is
downloaded into the Hugging Face cache on first use (clef-flash: 19 GB),
so this is enough to start:

```bash
mlx-decision run -m Cloudflare/clef-flash -s "My card was charged twice." --noul "Is it a billing problem?"
```

To fetch a model ahead of time, `mlx-decision download` shows a menu of
the supported models (or takes a repo id). In a terminal it first asks for
a folder to download into instead of the Hugging Face cache; the model
goes into a folder named after the repo inside it (e.g.
`~/Models/clef-flash`), which you then pass to `-m`, and an empty answer
keeps the cache. `--local-dir` names the model's folder directly and skips
the question. For other repo ids it reads the repository's config files
first and warns before downloading something no supported family can
load; private or gated repos need `HF_TOKEN`.

## Python

```python
import mlx_decision
from mlx_decision import Choice, Noul, Score

model = mlx_decision.load("Cloudflare/clef-flash")

ticket = (
    "Hi, I've been trying to connect my Stripe account for 3 days and the "
    "integration keeps failing. I'm losing sales. Please help ASAP."
)
result = model.decide(
    ticket,
    {
        "department": Choice(
            instructions="Which team should handle this",
            criteria={
                "billing": "Payment or subscription issues",
                "technical": "Bugs or integration problems",
                "sales": "Pricing or account questions",
            },
        ),
        "frustration": Score(
            instructions="How frustrated the customer appears",
            criteria=[
                "Calm, just stating facts",
                "Frustrated but civil",
                "Very angry, strong language",
            ],
        ),
        "is_urgent": Noul(instructions="The message conveys urgency or time-sensitivity"),
    },
)

print(result.answers["department"].choice)      # technical
print(result.answers["department"].confidence)  # 0.90
print(result.answers["frustration"].score)      # 1.29 (expected level, 0-based)
print(result.answers["is_urgent"].noul)         # 0.94 (probability of yes)
print(result.to_wire())                         # the response body, as JSON data
```

Questions can also be plain dicts in the API's wire format
(`{"type": "choice", "instructions": ..., "criteria": {...}}`), and
`model.decide_request(body)` takes a whole request body. Invalid requests
raise `mlx_decision.DecisionError`, whose `param` names the offending field
(building an invalid `Choice`, `Score` or `Noul` object directly raises
pydantic's `ValidationError` instead). `result.truncated` tells you whether
the state was cut to fit the model's input limit (16,384 tokens for
clef-flash). The answer types (`ChoiceAnswer`, `ScoreAnswer`,
`NoulAnswer`) and `Usage` can be imported from `mlx_decision`.

With the `images` extra, images go along as file paths, bytes, PIL images
or data URLs:

```python
result = model.decide(
    "Customer says checkout fails; screenshot attached.",
    {"shows_error": Noul(instructions="The screenshot shows an error message")},
    images=["screenshot.png"],
)
```

See [Images](#images) for how images count against the input limit.

`confidence` follows the definitions of the Jev API (choice: how far the top
probability sits above an even split; score: how concentrated the
probability is around the most likely level), not Clef's own top
probability.

## Command line

Ask questions about a state:

```bash
mlx-decision run -m Cloudflare/clef-flash \
  -s "My card was charged twice this month." \
  --noul "refund=The customer wants money back" \
  --choice "team=billing,technical,sales" \
  --score "anger=calm,annoyed,angry"
```

```
refund (noul): no, p(yes) 0.110
  ███

team (choice): billing, confidence 0.98
  billing    0.989  ████████████████████████
  technical  0.007
  sales      0.004

anger (score): 1.05 on 0-2, confidence 0.54
   0  0.128  ███                       calm
   1  0.695  █████████████████         annoyed
   2  0.177  ████                      angry

clef-flash · 291 input tokens
```

A score's answer is the expected level, weighted by the probabilities:
0 × 0.128 + 1 × 0.695 + 2 × 0.177 = 1.05 on a scale of 0 to 2.

The state can also come from `--state-file` or stdin (`--state-json` parses
it as JSON), and questions from a JSON file with `-q questions.json`
(a questions map or a whole request body). `--json` prints the exact
response body. `--image FILE` (repeatable) sends images along with the
state.

`mlx-decision --version` prints the installed version.

### Many states

`--states FILE` answers a JSON lines file with one request body per line
and prints one response body per line, in the same order. `-` reads
stdin. A line is the same body the server takes, so it can bring its own
questions; questions from `-q` or the shorthand flags apply to lines
without them:

```bash
mlx-decision run -m Cloudflare/clef-flash -q questions.json --states tickets.jsonl > answers.jsonl
```

```
{"state": "My card was charged twice this month."}
{"state": {"ticket": 4711, "text": "The app crashes on login."}}
{"state": "Is this spam?", "questions": {"spam": {"type": "noul", "instructions": "Is it spam?"}}}
```

Lines can carry images as file paths or data URLs
(`{"state": "...", "images": ["shots/4711.png"]}`); `--image` flags apply
to lines without their own. A line that cannot be answered yields an
error body with its line number, and the run goes on:

```
{"error": {"message": "state is required", "type": "invalid_request_error", "param": "state"}, "line": 3}
```

## Interactive: `chat`

`mlx-decision chat` loads the model once and answers one state after
another: type or paste a state, finish it with a blank line, and quit with
Ctrl-D. Lines can be edited, Up recalls earlier states, and a toolbar at
the bottom shows the keys. Questions come from the same flags and files as
in `run`:

```bash
mlx-decision chat -m Cloudflare/clef-flash --choice "team=billing,technical,sales"
```

Without any questions, `chat` first asks for them: the type from a menu
(arrow keys or the number, then Enter), an id (Enter takes the one
suggested), the instructions, and then the options of a choice (`option`
or `option: description`) or the levels of a score (lowest first), one per
line, ending with a blank line. Each question is checked against the model
as soon as it is complete, and shown as understood.

```
$ mlx-decision chat -m Cloudflare/clef-flash
clef-flash loaded, 0 questions.
No questions yet: build the first one (Ctrl-C cancels).
 Type:
   >  1. choice  pick one of several options
      2. score   rate on ordered levels
      3. noul    a yes/no question
id: team
instructions (optional): Which team should handle this?
options, one per line as 'option' or 'option: description' (blank line ends):
  billing: payments, invoices, refunds
  technical
  sales

added:
  team (choice): Which team should handle this?
    billing: payments, invoices, refunds
    technical
    sales
 Next:
      1. Start answering states
   >  2. Add another question
 Type:
      1. choice  pick one of several options
      2. score   rate on ordered levels
   >  3. noul    a yes/no question
id: refund
instructions: The customer wants money back
added:
  refund (noul): The customer wants money back
 Next:
   >  1. Start answering states
      2. Add another question
Type a state and finish it with a blank line. /help lists the commands, Ctrl-D quits.
state> My card was charged twice this month.

team (choice): billing, confidence 0.93
  billing    0.951  ███████████████████████
  technical  0.028  █
  sales      0.021  █

refund (noul): no, p(yes) 0.030
  █

clef-flash · 225 input tokens

state> /save questions.json
saved 2 questions to questions.json
```

Between states, commands change the questions, whether they came from the
builder, flags or a file. `/edit` and `/remove` without an id offer a menu
of the questions.

| Command | |
|---|---|
| `/list` | show the questions |
| `/add` | build another question |
| `/edit [ID]` | change a question; the current values are filled in |
| `/remove [ID]` | remove a question |
| `/save FILE` | write the questions as a file that `-q` reads |
| `/image FILE`, `/image clear` | attach an image to the following states, or drop them (`--image` attaches at start) |
| `/help`, `/quit` | |

A state that itself starts with `/` is typed as `//`. `chat` needs a
terminal; scripts use `run --states`. With `--json`, the response bodies
go to stdout and can be redirected while the prompts stay on screen.

## Server

```bash
mlx-decision server -m Cloudflare/clef-flash --port 8000
```

Serves `POST /v1/systemone`, compatible with the Jev API's request and
response bodies, plus `GET /v1/models` and `GET /health`. Clients written
for that API work by changing the base URL, for example with its Python
SDK:

```bash
TYPESAFE_BASE_URL=http://127.0.0.1:8000 TYPESAFE_API_KEY=unused python my_script.py
```

The server binds to localhost by default (`--host` to change), ignores the
request's `model` field, answers one request at a time (others queue),
returns `422` with the API's error body for invalid requests, and sets
`X-MLX-Decision-Truncated: true` when the state was truncated. Request
bodies larger than 64 MB get `413`.

Images go in the request's `images` list as base64 data URLs
(`data:image/png;base64,...`; PNG, JPEG or WebP). The server reads no files
and fetches no URLs, so anything else is refused with `422` naming the
image (`images.0`). With the SDK, pass them as
`extra_body={"images": [...]}`.

To require a key, start it with `--api-key KEY` (or set
`MLX_DECISION_API_KEY`). Requests to `/v1/*` then need
`Authorization: Bearer KEY`, which is what the SDK sends from
`TYPESAFE_API_KEY`; others get `401`. `/health` stays open.

## Smaller models: `convert`

```bash
# 8-bit: half the memory, answers within the parity tolerance
mlx-decision convert -m Cloudflare/clef-flash -q                 # -> clef-flash-q8

# mixed precision: bits go where the answers are most sensitive
mlx-decision convert -m Cloudflare/clef-flash --target-bits 5    # -> clef-flash-mq5
```

Without `-o` the folder is named after the model plus `-q<bits>`,
`-mq<target>` or `-mlx` (no quantization). A converted folder loads like
the original (`mlx_decision.load("clef-flash-q8")`). Only the text
backbone is quantized; the decision head and the vision tower (0.85 GB)
stay as released. With `--target-bits` the converter
first measures, on a built-in calibration set, how far the answers move
when each block of the model is quantized alone (about 6 minutes for
clef-flash), then gives the sensitive blocks more bits within the average
you asked for.

Measured on the 50 text requests of the parity set (139 questions)
against the unquantized model:

| Model | Disk | Peak memory | Mean / max probability difference | Changed answers |
|---|---|---|---|---|
| bf16 (as released) | 17.8 GB | 19.0 GB | - | - |
| 8-bit (`-q`) | 9.1 GB | 11.2 GB | 0.001 / 0.014 | 0 |
| mixed 5 (`--target-bits 5`) | 6.0 GB | 8.1 GB | 0.003 / 0.070 | 0 |
| mixed 4 (`--target-bits 4`) | 4.9 GB | 7.1 GB | 0.006 / 0.084 | 1 |
| uniform 4-bit (`-q --bits 4`) | 4.9 GB | 7.1 GB | 0.014 / 0.212 | 9 |

Recommendation: use the released bf16 model if 19 GB fit; otherwise 8-bit;
on smaller Macs `--target-bits 5`, then `--target-bits 4`. Uniform 4-bit
loses noticeably more than mixed precision of the same size. Quantization
saves memory, not time: a decision is one compute-bound pass, and quantized
models are 10-20% slower.

## Speed: `benchmark`

```bash
mlx-decision benchmark -m Cloudflare/clef-flash
```

Reports load time, peak memory, and median / p95 latency per request across
state lengths and numbers of questions. Median latency per request for
clef-flash on an Apple M5 Pro (20-core GPU) with 64 GB:

| Input tokens | Questions | bf16 | 8-bit | uniform 4-bit |
|---|---|---|---|---|
| 394 | 1 | 0.25 s | 0.27 s | 0.25 s |
| 1,552 | 5 | 0.84 s | 1.07 s | 1.03 s |
| 4,149 | 1 | 2.41 s | 2.89 s | 2.76 s |
| 6,103 | 20 | 3.63 s | 4.33 s | 4.13 s |
| 16,384 | 20 | 10.7 s | 12.1 s | 11.4 s |
| Peak memory | | 19.0 GB | 11.2 GB | 7.1 GB |

Latency grows linearly with input length, at about 1,550-1,850 tokens per
second. All numbers in this README were measured on that machine.

## Images

Images need the `images` extra. The Python API takes file paths, bytes,
PIL images or data URLs; `run` and `chat` take files (`--image`,
`/image`); the server takes data URLs only. Files, bytes and data URLs
must be PNG, JPEG or WebP, at most about 89 million pixels (Pillow's
decompression-bomb limit). Each image is resized the way
Cloudflare's processor does it (sides rounded to multiples of 32, at least
256x256 worth of pixels) and costs one token per 32x32 pixels.

Images larger than 2 megapixels (2^21 pixels, about 2,048 tokens) are
shrunk to that size, keeping their aspect ratio. On test pictures with
small text (a log window, an A4 invoice scan, a settings page) and photos
of up to 18 MP, this changed no answer, while 0.5 MP and below misread
small text. `--max-image-mp` on `run`, `chat` and `server` (or
`load(..., max_image_pixels=...)` in Python) sets another cap; `0` lifts it
to the processor's own maximum (16.7 MP), which is what Cloudflare's
reference does (in Python, `max_image_pixels=None`). Time per request with
a short state and three questions (text only: 0.23 s):

| Image | Tokens | bf16 | 8-bit |
|---|---|---|---|
| 256x256 or smaller | 64 | 0.27 s | 0.29 s |
| 640x480 | 300 | 0.47 s | 0.51 s |
| 1024x768 | 768 | 0.84 s | 1.03 s |
| 1920x1080, or larger (shrunk to 2 MP) | about 2,048 | 2.3 s | 3.0 s |
| 4000x3000 with `--max-image-mp 0` | 11,750 | 36 s | 39 s |

Images count against the 16,384-token input limit together with the
questions: the state is cut first, and a request whose images and
questions alone do not fit fails with an error on `images`, before any
image is decoded. The vision tower (0.85 GB) loads on the first image
request, so text-only use costs no extra memory.

## Supported models

| Family | Models | Input | Notes |
|---|---|---|---|
| Clef | `Cloudflare/clef-flash` | text, images | videos are rejected |

On a 63-request test set (13 of them with one to three images), clef-flash
on MLX matches Cloudflare's PyTorch reference token for token and within
0.035 per probability (mean 0.001; with images 0.015, mean 0.002), with no
changed answers.

### Accuracy (preliminary)

Preliminary results: clef-flash (bf16, this package) and Jev, TypeSafe
AI's hosted model, on five public benchmarks, 500 test examples each (fixed sample;
larger runs and quantized models will follow), with the same questions
and option descriptions for both. Accuracy / macro-F1 in percent; ECE is
the expected calibration error of the top probability (lower is better).
In brackets: macro-F1 published by Cloudflare (clef-flash / Jev), from
their own, unpublished prompts.

| Benchmark | Options | clef-flash acc / F1 / ECE | Jev acc / F1 / ECE | Published F1 or acc |
|---|---|---|---|---|
| AG News (topic) | 4 | **91.4** / **91.5** / 2.5 | 86.4 / 86.4 / 9.5 | - |
| DAIR Emotion | 6 | 60.0 / 54.6 / 19.4 | **62.2** / **56.2** / 26.0 | - |
| ANLI r1-r3 (entailment) | 3 | 58.2 / 57.9 / 14.1 | **71.6** / **71.9** / 11.5 | 59.1 / 74.8 |
| BANKING77 (intent) | 77 | **96.0** / **95.8** / 3.5 | 80.2 / 79.4 / 8.2 | 90.9 / 79.7 |
| MMLU (accuracy) | 4 | **93.0** / 93.0 / 6.0 | 91.6 / 91.6 / 3.7 | 91.8 / 91.7 |

clef-flash is stronger at classification with many or well-described
labels (BANKING77, AG News) and Jev at adversarial entailment (ANLI). Both
are poorly calibrated on Emotion, whose labels overlap. Reproduce the
local numbers with `scripts/accuracy_benchmark.py local` (its docstring
shows how to save the datasets); the Jev column needs a Jev account and is
not reproducible with this repository alone.

## Development

```bash
make setup   # editable install with dev tools
make test    # unit tests; parity tests run when the weights are found
make lint
```

Tests that need clef-flash look for it in `$MLX_DECISION_MODELS/clef-flash`,
then in the Hugging Face cache, and are skipped otherwise. The parity
fixtures are rebuilt with `scripts/make_parity_set.py` and
`scripts/make_parity_reference.py` (the latter runs Cloudflare's reference
with PyTorch).

## Relation to Cloudflare and TypeSafe AI

mlx-decision is an independent open-source project, not affiliated with or
endorsed by Cloudflare or TypeSafe AI. Its server speaks a request and
response format compatible with TypeSafe AI's public Jev API. clef-flash
is Cloudflare's model, used under its Apache-2.0 licence; this package
downloads it from Hugging Face and does not ship its weights.

## Licence

MIT, see `LICENSE`. The Qwen3.5 model code is adapted from
[mlx-lm](https://github.com/ml-explore/mlx-lm) (MIT, Apple Inc.); the Clef
input encoding and head are ported from Cloudflare's release (Apache-2.0);
the image preprocessing, vision tower and image positions are ported from
[transformers](https://github.com/huggingface/transformers) (Apache-2.0).
Their licence texts are in `LICENSES/`. The test photos are public domain
or CC0 (`tests/parity/images/SOURCES.md`). Model weights keep their own
licence (clef-flash: Apache-2.0).
