Metadata-Version: 2.4
Name: xiaomi-lens-mnn
Version: 0.1.2
Summary: Xiaomi-Lens-0 referring expression grounding + segmentation on MNN, no torch
Author: yunfeng
License: Apache-2.0
Project-URL: Homepage, https://huggingface.co/yunfengwang/Xiaomi-Lens-0-MNN-fp16
Project-URL: Model, https://huggingface.co/yunfengwang/Xiaomi-Lens-0-MNN-fp16
Project-URL: Mirror, https://modelscope.cn/models/yunfeng/Xiaomi-Lens-0-MNN-fp16
Keywords: mnn,xiaomi-lens,vision-language,segmentation,on-device,grounding
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mnn<3.7,>=3.6.1
Requires-Dist: numpy>=1.24
Requires-Dist: pillow>=9.5
Requires-Dist: jinja2>=3.1
Requires-Dist: tokenizers>=0.15
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

# xiaomi-lens-mnn

Run **Xiaomi-Lens-0** (4.6 B vision-language grounding + segmentation model) on
[MNN](https://github.com/alibaba/MNN) with one command. No torch, no transformers,
no 8.6 GB bf16 checkpoint — the CLI downloads three MNN graphs plus the embedding
table and runs the vision encoder, the Qwen3.5 hybrid decoder and the pixel-mask
decoder entirely in MNN (CPU). Choose fp16 (10.5 GB), int8 (6.2 GB) or int4
(4.1 GB, 4-bit decoder + 8-bit vision).

```bash
uvx xiaomi-lens-mnn --image cow.jpg \
  --describe 'a large brown cow standing in the grass'
```

Output:

```
[xiaomi-lens-mnn] fp16 result
  prompt tokens : 255 (216 image, grid 24x36)
  completion    : 32 tokens
  bboxes        : [[462, 302, 854, 754]]
  masks         : 1
  timings       : vision 1.80s, preprocess 0.04s, prefill 70.11s, decode 6.74s, mask 0.85s, total 79.54s
  overlay       : /path/lens-out/overlay.png

<bbox>[462, 302, 854, 754]</bbox><mask><|mask|></mask>
```

### Prompting

`--describe EXPR` builds the instruction the model expects for grounding:

```
Locate all instances in the image that match the description "EXPR". Report bbox coordinates and masks.
```

Pass the same text to `--prompt` if you prefer (or if you are on 0.1.0). The
instruction shape matters: a bare phrase such as `--prompt 'a large brown cow'`
is answered as a caption and comes back **without** a bbox or mask. Xiaomi-Lens-0
also answers free-form questions, so `--prompt` itself is left unmodified —
`--describe` is the safe way to ask for boxes and masks.

Files written to `lens-out/`: `overlay.png` (coloured masks on the input image),
`mask_0.png`, `masks.npy` (float32 logits on the 384×576 model canvas, `K×H×W`)
and `result.json`. `result.json` also records `bboxes` mapped back to the
original image (0–1000), the raw `bboxes_canvas`, `content_box` and
`original_size`.

## Install

`uvx` runs it without installing anything globally:

```bash
# first run downloads ~10.5 GB (fp16) into ~/.cache/xiaomi-lens-mnn/fp16
uvx xiaomi-lens-mnn --image photo.jpg --describe "the red car on the left"

# smaller / faster download (6.2 GB), slight mask degradation
uvx xiaomi-lens-mnn --image photo.jpg --precision int8 \
  --describe "the red car on the left"

# smallest download (4.1 GB), 4-bit decoder, int8 vision
uvx xiaomi-lens-mnn --image photo.jpg --precision int4 \
  --describe "the red car on the left"

# ModelScope mirror (recommended from mainland China)
uvx xiaomi-lens-mnn --image photo.jpg --source ms --describe "the red car on the left"

# fully offline after downloading the repo
uvx xiaomi-lens-mnn --image photo.jpg --model-dir ./Xiaomi-Lens-0-MNN-fp16 \
  --describe "the red car on the left"
```

Classic install: `pip install xiaomi-lens-mnn` then `xiaomi-lens-mnn --help`.
Requires Python 3.10–3.14 and a platform with an MNN wheel (macOS arm64/x86_64,
Linux aarch64/x86_64, Windows x86_64).

## CLI

```
xiaomi-lens-mnn --image IMAGE [--describe EXPR | --prompt PROMPT]
                [--precision fp16|int8|int4] [--source auto|hf|ms] [--model-dir DIR]
                [--cache-dir DIR] [--threads N] [--max-new N] [--no-think]
                [--out-dir DIR] [--json]
```

| flag | default | meaning |
| --- | --- | --- |
| `--image` | required | input image, any format Pillow reads |
| `--describe` | — | build the grounding instruction for this expression (bbox + mask) |
| `--prompt` | cow grounding demo | free-form instruction or question, passed through unchanged |
| `--precision` | `fp16` | `fp16`, `int8`, or `int4` (4-bit decoder + int8 ViT/mask) |
| `--source` | `auto` | `hf`, `ms`, or auto (try HF, fall back to ModelScope) |
| `--model-dir` | — | already downloaded repo directory; skips the network |
| `--cache-dir` | `~/.cache/xiaomi-lens-mnn` | download cache |
| `--threads` | `15` | MNN CPU threads |
| `--max-new` | `128` | maximum generated tokens |
| `--no-think` | off | disable the `<think>` block (shorter, faster, less reasoning) |
| `--out-dir` | `lens-out` | output directory |
| `--json` | off | print the full result JSON |

### Python API

```python
from PIL import Image
from xiaomi_lens_mnn import LensMNN, grounding_prompt
from xiaomi_lens_mnn.download import ensure_model

model_dir = ensure_model("fp16", "auto")       # cached after the first call
runner = LensMNN(model_dir, threads=15)
result = runner.run(
    Image.open("cow.jpg"), grounding_prompt("the large brown cow"), out_dir="out")
print(result["bboxes"], result["masks"], result["timings"])
```

`grounding_prompt("...")` is the same expansion as `--describe`; passing a bare
phrase to `runner.run` returns a caption instead of a mask.

## Speed

Measured on an Apple M5 Pro (48 GB), CPU backend, 15 threads, same 586×394 image,
255-token prompt (216 image tokens), 32 generated tokens, one mask. Lower is
better. Full data: `benchmarks.json` in the weight repos.

| stage | torch fp32 (HF) | ONNX Runtime | MNN fp16 | MNN int8 |
| --- | ---: | ---: | ---: | ---: |
| vision (24×36 grid) | 0.74 s | 1.41 s | 1.80 s | 1.84 s |
| LLM prefill (255 tok) | 24.78 s | 84.46 s | 70.11 s | 64.46 s |
| LLM decode (32 tok) | 600.07 s | 10.36 s | 6.74 s | 6.10 s |
| pixel mask (1 mask) | 0.35 s | 0.15 s | 0.85 s | 0.53 s |
| **end to end** | **625.9 s** | **96.4 s** | **79.5 s** | **73.0 s** |
| speed-up vs torch | 1× | 6.5× | 7.9× | 8.6× |
| weights on disk | 18.5 GB (fp32) | 17.6 GB | 10.5 GB | 6.2 GB |
| decode per token | 18.75 s | 0.324 s | 0.211 s | 0.191 s |

The MNN decoder graph is **incremental** (one token per session run), so its
prefill is a 255-step loop and is slower than torch's parallel prefill. Vision
and mask are also a little slower in MNN than in torch/ONNX. The win is decode:
MNN is 89× faster per token than the official HF CPU path, because torch's
GatedDeltaNet `conv1d` falls back to the slow im2col kernel there. End to end,
MNN fp16 is 7.9× and int8 8.6× faster than torch while using 44 % / 67 % less
disk. `int8` is ~8 % faster than `fp16`, with a small mask drift (see below).

ONNX Runtime sits between them: its fp32 decoder is numerically exact but pays a
large MatMul cost on CPU; its fp16 ViT/mask graphs are fast (mask 0.15 s).

### MNN precision head-to-head

All three MNN builds re-measured back-to-back in one idle session (same machine,
image, prompt, 32 generated tokens, one mask). Absolute times drift 10–20 %
between sessions with thermal state; compare within this table, not against the
older one above.

| stage | fp16 | int8 | int4 |
| --- | ---: | ---: | ---: |
| vision | 1.55 s | 1.34 s | 1.34 s |
| LLM prefill (255 tok) | 51.21 s | 47.45 s | 46.61 s |
| LLM decode (32 tok) | 4.81 s | 4.85 s | 4.63 s |
| pixel mask | 0.51 s | 0.37 s | 0.34 s |
| **end to end** | **58.1 s** | **54.0 s** | **52.9 s** |
| decode per token | 0.150 s | 0.152 s | 0.145 s |
| weights on disk | 10.5 GB | 6.2 GB | 4.1 GB |

`int4` is ~2 % faster than `int8` and ~9 % faster than `fp16`. Decode is nearly
identical for int8/int4: the 4-bit kernels save memory bandwidth but pay it back
in dequantization. The real win is download size — 4.1 GB instead of 6.2 GB.

## Precision / verification

| check | fp16 | int8 | int4 |
| --- | --- | --- | --- |
| generated text / bbox vs HF | identical | identical | bbox digits shift ≤ 1.6 % of image width |
| pixel-mask IoU vs HF (cow) | 1.0000 | 0.9970 | 0.9964 |
| 5-image mask IoU vs fp16, mean / worst | — | 0.9972 / 0.9945 | 0.9863 / 0.9546 |
| 5-image bbox IoU vs fp16, mean / worst | — | 1.000 / 1.000 | 0.979 / 0.936 |
| peak RSS during a 64-token run | 18.4 GB | 19.1 GB | 20.2 GB |
| decoder logits max / mean abs error (8 steps) | 1.8e-3 / 3.6e-5 | 0.4268 / 0.3519 | not measured |
| ViT merged max / mean abs error | 7.2e-3 / 3.6e-5 | 1.095 / 2.8e-3 | same as int8 |
| mask logits max / mean abs error | 6.0e-3 / 2.2e-3 | 3.143 / 0.2852 | same as int8 |

`int8` is weight-only 8-bit block quantization (`--weightQuantBits 8
--weightQuantAsymmetric --weightQuantBlock 64`). The ViT and mask decoder show
visible numerical drift but the final binary mask is still within IoU 0.997.

`int4` is **mixed precision**: the 4.6 B decoder uses 4-bit block-wise
asymmetric weights (`--weightQuantBits 4 --weightQuantAsymmetric
--weightQuantBlock 64`), while the ViT and mask decoder stay int8. A pure 4-bit
build saves only ~0.2 GB more but degrades masks: 5-image mean IoU 0.906, worst
0.689 (Mona Lisa). 4-bit ViT alone was the main culprit (0.841 on that image at
block 64, 0.969 with block 32); keeping vision in 8 bits is the better trade.

**int4 reduces download/disk, not runtime RAM.** MNN's CPU backend materializes
weights into fp32 working copies, so a 4-bit decoder expands ~8× in RAM and the
measured peak is slightly higher than int8 (20.2 vs 19.1 GB), not lower.

Use `fp16` when mask boundaries must be exact, `int4` when the 4.1 GB download
or repo size matters.

## How it works

Three MNN graphs:

1. **`lens_vit.mnn`** — Qwen3.5 ViT (24 blocks, 1024 hidden, patch 16, merge 2).
   Position-embedding interpolation and 2-D rotary cos/sin are computed in numpy
   outside the graph, so arbitrary patch grids work. Outputs pre-merge features
   (for the mask decoder) and merged image features (for the LLM).
2. **`lens_llm_decode.mnn`** — one-token Qwen3.5-4B decoder step with explicit
   KV / conv / recurrent states. 24 GatedDeltaNet layers + 8 full-attention
   layers, hidden 2560, vocab 248,078 (tied lm_head split into two 124,039-wide
   MatMuls). M-RoPE positions are computed in numpy with interleaved
   `[11, 11, 10]` sections. The `<|mask|>` hidden state is captured while
   decoding.
3. **`lens_mask.mnn`** — pixel-mask decoder over the 24×36 patch canvas:
   mask embedding, patch self-attention, channel reduce, two transposed
   convolutions + 4× bilinear upsampling, output 32 masks at 384×576.

The package itself has no torch/transformers dependency. It ports the Qwen2-VL
smart-resize + patchify (BICUBIC, factor 32, min 65,536 / max 8,388,608 pixels),
the official chat template, the tokenizer (`tokenizers`), M-RoPE and the vision
position/rotary tables to numpy/Pillow. Parity with the official processor is
byte-exact for token ids and < 1e-2 max for pixel values (fp16 rounding).

## Weights

| repo | precision | files |
| --- | --- | --- |
| [Xiaomi-Lens-0-MNN-fp16](https://huggingface.co/yunfengwang/Xiaomi-Lens-0-MNN-fp16) | fp16 | ViT 633 MB, mask 159 MB, decoder 7.8 GB, embeddings 1.2 GB |
| [Xiaomi-Lens-0-MNN-int8](https://huggingface.co/yunfengwang/Xiaomi-Lens-0-MNN-int8) | int8 | ViT 337 MB, mask 85 MB, decoder 4.2 GB, embeddings 1.2 GB |
| [Xiaomi-Lens-0-MNN-int4](https://huggingface.co/yunfengwang/Xiaomi-Lens-0-MNN-int4) | int4 decoder + int8 vision | ViT 337 MB, mask 85 MB, decoder 2.4 GB, embeddings 1.2 GB |
| [Xiaomi-Lens-0-onnx](https://huggingface.co/yunfengwang/Xiaomi-Lens-0-onnx) | fp32 / fp16 | ONNX graphs for re-conversion |

ModelScope mirror: `yunfeng/Xiaomi-Lens-0-MNN-fp16` / `...-int8` / `...-int4`.
The first run downloads `manifest.json`, the three graphs and the fp16 embedding
table into `~/.cache/xiaomi-lens-mnn/<precision>/`. Downloads resume after
interruption. No token is required; public repos.

## Limitations

- **Fixed 576×384 input canvas** (24×36 patches). Every image is fitted onto
  that canvas: near-3:2 images are resized directly, others are centre
  letterboxed with neutral grey and boxes/masks are mapped back to the original
  image. High-resolution detail is therefore reduced to ~0.22 MP; crop first if
  you need small-object accuracy. Re-export `lens_mask` for other canvas sizes.
- The decoder graph is incremental; prefill is sequential. See the speed table.
- CPU only (MNN Metal is not enabled; the release was verified on CPU).
- An image + one prompt per process. The states are reset for each `run()`.
- Grounding needs an instruction that asks for bbox + mask; use `--describe`
  (or `grounding_prompt`) rather than a bare object name.
- `int4` does not lower peak RAM (MNN expands quantized weights to fp32);
  it lowers download and disk only. Peak measured 20.2 GB vs 19.1 GB for int8.
- `int8` mask boundaries drift slightly (IoU 0.997 vs fp16); `int4` drifts more
  on some images (worst 0.955 over five demo images). Prefer fp16 for
  precise segmentation.
- The mask decoder is trained for the model's `<|mask|>` token; the CLI overlays
  only answer masks by default (thinking masks are counted but not drawn).

## License

Apache-2.0, following the Xiaomi-Lens-0 release. The MNN graphs are derivative
weights of [SeerRay-Lab/Xiaomi-Lens-0](https://huggingface.co/SeerRay-Lab/Xiaomi-Lens-0).
