Metadata-Version: 2.5
Name: mlx-beam-vision
Version: 0.1.0a5
Summary: Vision towers for mlx-beam: images in the prompt, served by the checkpoint's own tower.
Project-URL: Homepage, https://p4ik.github.io/mlx-beam/
Project-URL: Repository, https://github.com/p4ik/mlx-beam
Author: the mlx-beam contributors
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: apple-silicon,inference,mlx,vision,vlm
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: mlx-beam
Requires-Dist: mlx>=0.32; sys_platform == 'darwin'
Requires-Dist: numpy
Requires-Dist: pillow>=10
Requires-Dist: transformers>=5.15
Provides-Extra: dev
Requires-Dist: mlx[cpu]>=0.32; (sys_platform == 'linux') and extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# mlx-beam-vision

The towers that let [mlx-beam](https://github.com/p4ik/mlx-beam) see. Install
it next to the engine (`pip install mlx-beam[vision]`, or this package by
name) and a checkpoint that ships a vision tower serves images: an
`image_url` part with a `data:` URL in a chat request, the checkpoint's own
processor renders the template, the tower encodes the image, and the
engine's prefill takes the image's features in place of the placeholder
tokens - beside text-only requests in the same batch, with the prefix cache
keyed on the image's digest. Without this package the engine answers image
input with a 400 that names the `vision` extra.

Towers vendored from [mlx-vlm](https://github.com/Blaizzy/mlx-vlm) (MIT,
see `mlx_beam_vision/_vendor/VENDORED.md`):

| Family | Models | Notes |
|---|---|---|
| `qwen3_vl` | Qwen3-VL, Qwen3.5, Qwen3.8 | DeepStack applied: the tower's intermediate features are added after the text model's first layers |
| `mistral3` | Mistral Small 3.x (Pixtral tower) | the image's rows are separate spans (`[IMG_BREAK]` between them) |
| `gemma4` | Gemma 4 E2B, E4B | image only; audio is not served yet. The 12B (`gemma4_unified`, encoder-free, its image tokens attending bidirectionally in the text model) is refused with that reason until the text trunk takes such a mask |
| `muse_glimmer` | Muse Glimmer | the projector's norm matches the text model's input norm |
| `granite4_vision` | Granite Vision 4.1 | AnyRes tiles, window Q-Former projectors; nothing at the embedding, every projector's features added ahead of its text layer |

Not yet: audio and video input, quantized towers (the towers load at model
precision; a package that ships a quantized tower is refused with a
message). An image above 32 megapixels is refused before it is decoded.

Every family is exercised in the tests on a tiny random tower against a
direct forward; the real towers and processors are validated on Apple
silicon with the checkpoints themselves.

`/health.vision` reports the family, the tower, the processor and the
feature cache (encoder outputs by image digest, bounded in bytes).
