Metadata-Version: 2.4
Name: mosaix
Version: 0.1.0
Summary: Maximum-throughput video inference for YOLO and any vision model, via batched gutter-padded mosaic tiling.
Project-URL: Homepage, https://github.com/BARKEM-Cognitive-Industries/mosaix
Project-URL: Documentation, https://github.com/BARKEM-Cognitive-Industries/mosaix/tree/main/docs
Project-URL: Repository, https://github.com/BARKEM-Cognitive-Industries/mosaix
Project-URL: Issues, https://github.com/BARKEM-Cognitive-Industries/mosaix/issues
Project-URL: Changelog, https://github.com/BARKEM-Cognitive-Industries/mosaix/blob/main/CHANGELOG.md
Author: mosaix contributors
License: MIT
License-File: LICENSE
Keywords: acceleration,batch,computer-vision,gpu,inference,mosaic,object-detection,onnx,real-time,throughput,tiling,ultralytics,video,yolo
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: >=3.9
Requires-Dist: numpy>=1.21
Requires-Dist: opencv-python>=4.5
Provides-Extra: all
Requires-Dist: insightface>=0.7; extra == 'all'
Requires-Dist: onnxruntime-gpu>=1.15; extra == 'all'
Requires-Dist: transformers>=4.40; extra == 'all'
Requires-Dist: ultralytics>=8.1; extra == 'all'
Provides-Extra: depth
Requires-Dist: transformers>=4.40; extra == 'depth'
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: psutil>=5.9; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Provides-Extra: face
Requires-Dist: insightface>=0.7; extra == 'face'
Requires-Dist: onnxruntime-gpu>=1.15; extra == 'face'
Provides-Extra: onnx
Requires-Dist: onnxruntime-gpu>=1.15; extra == 'onnx'
Provides-Extra: onnx-cpu
Requires-Dist: onnxruntime>=1.15; extra == 'onnx-cpu'
Provides-Extra: ultralytics
Requires-Dist: ultralytics>=8.1; extra == 'ultralytics'
Description-Content-Type: text/markdown

# mosaix

**Maximum-throughput video inference for YOLO — and any vision model.**

`mosaix` turns an ordinary object detector into a firehose. On a single laptop RTX 4060
(8 GB) it pushes **YOLO11n to ~1735 FPS end-to-end / 2178 FPS inference-only** on real,
multi-minute video — decode included — without exotic hardware, custom CUDA, or model
retraining. Point it at a model and a video; it gives you per-frame detections and the
throughput number.

```python
import mosaix

pipe = mosaix.VideoPipeline.from_model("yolo11n.pt")
result = pipe.run("long_video.mp4")

print(f"{result.fps:.0f} FPS end-to-end, {result.total_detections} detections")
for frame in result:
    for det in frame.detections:
        print(frame.index, det.cls, det.conf, det.xyxy)
```

---

## How it hits 1000+ FPS

A detector spends most of its time on per-image fixed overhead (launch, letterbox,
NMS, host↔device copies), not on the pixels that matter. `mosaix` amortises that
overhead across many frames at once:

```
                                 one GPU forward pass
        ┌───────────────────────────────────────────────────────────┐
 frames │  ┌────┬────┬────┐   ┌────┬────┬────┐        ┌────┬────┬────┐ │
 0..287 │  │ f0 │ f1 │ f2 │   │ f9 │f10 │f11 │  ...   │f279│f280│f281│ │
        │  ├────┼────┼────┤   ├────┼────┼────┤        ├────┼────┼────┤ │
        │  │ f3 │ f4 │ f5 │   │f12 │f13 │f14 │        │... │... │... │ │
        │  ├────┼────┼────┤   ├────┼────┼────┤        ├────┼────┼────┤ │
        │  │ f6 │ f7 │ f8 │   │f15 │f16 │f17 │        │... │...│f287 │ │
        │  └────┴────┴────┘   └────┴────┴────┘        └────┴────┴────┘ │
        │     mosaic 0           mosaic 1      ...        mosaic 31     │
        └───────────────────────────────────────────────────────────┘
                  grid = 9 cells       ×       batch = 32 mosaics
                          =  288 frames per single forward pass
```

1. **Downscale** every frame to a small cell (default `426×240`, i.e. "240p").
2. **Mosaic** `grid` cells (default 9 → a 3×3 grid) into one image, with a blank
   **gutter** between cells.
3. **Batch** `batch` mosaics (default 32) into a single model forward pass — so each
   pass covers `grid × batch = 288` frames.
4. **Remap** every detection back to the exact frame it came from.

## Install

```bash
pip install mosaix                 # core (numpy + opencv)
pip install "mosaix[ultralytics]"  # + YOLO / RT-DETR support
pip install "mosaix[onnx]"         # + ONNX Runtime (GPU)
pip install "mosaix[all]"          # everything
```

GPU inference needs a CUDA-enabled PyTorch (for the Ultralytics backend) or
`onnxruntime-gpu` (for the ONNX backend) — install those per your CUDA version.

---

## Tested models & benchmarks

mosaix is built around bounding-box detectors, but the adapter layer also runs
classifiers, taggers, depth, dense-pose and face models. The table below is **measured
on this repo's models** so you know what to expect *before* you try one.

**Test rig:** NVIDIA RTX 4060 Laptop (8 GB), FP16 on CUDA, OpenCV decode (no `decord`).
**Clip:** *The Monkey Business Illusion* — 854×480, ~30 fps, many people (a non-private,
reproducible stand-in for crowded real footage). Each model ran in its **own process**
(uncontested), sweeping configs `1×1` (no tiling), `4×32` and `9×32` (`grid×batch`).
600 frames per config for detectors, fewer for heavy whole-image nets.

> These are **relative, apples-to-apples** numbers across families on a short 480p clip
> — they are decode-bound (note the low GPU-util), not peak. For tuned single-model peak
> (YOLO11n hits ~1735 e2e / 2178 infer FPS), see [`docs/performance.md`](docs/performance.md).
> Install `decord` and use a 240p source to get there.

### Throughput

`e2e` = decode→mosaic→infer→remap (what you get); `infer` = GPU forward only.
`tiling×` = best e2e ÷ untiled (`1×1`) e2e — the speedup mosaicking buys.
VRAM is the Torch allocator peak; ONNX/onnxruntime memory isn't visible to it (`—`).

| Model | Task | Backend | CUDA e2e | CUDA infer | tiling× | best `g×b` | VRAM | CPU e2e |
|---|---|--:|--:|--:|--:|:--:|--:|--:|
| FastSAM-s | segment | ultralytics | **308** | 494 | 5.5× | 4×32 | 1187 MB | 43 |
| omniparser_icon_detect | detect (UI icons) | ultralytics | 250 | 360 | 4.5× | 4×32 | 684 MB | 26 |
| yolov12n-face | detect (face) | ultralytics | 235 | 473 | 6.7× | 9×32 | 413 MB | 94 |
| yolo11m | detect | ultralytics | 231 | 355 | 4.3× | 4×32 | 684 MB | 23 |
| yolo11m-pose | pose | ultralytics | 223 | 349 | 4.7× | 4×32 | 646 MB | 21 |
| rtdetr-l | detect (DETR) | ultralytics | 223 | 271 | **8.5×** | 4×32 | 1268 MB | 4.0 |
| FastSAM-x | segment | ultralytics | 220 | 265 | 4.7× | 4×32 | 1803 MB | 4.6 |
| yolo11n-cls | classify | ultralytics | 207 | 247 | 2.6× | 1×32 | 123 MB | 75 |
| yolov12m-face | detect (face) | ultralytics | 198 | 304 | 5.0× | 4×32 | 644 MB | 22 |
| yolo11x-pose | pose | ultralytics | 191 | 244 | 5.8× | 4×32 | 1242 MB | 4.7 |
| yolov12l-face | detect (face) | ultralytics | 190 | 268 | 8.0× | 4×32 | 770 MB | 16 |
| yolo11m-seg | segment | ultralytics | 189 | 268 | 4.4× | 4×32 | 1186 MB | 16 |
| yolo11n | detect | ultralytics | 186 | 309 | 3.0× | 4×32 | 212 MB | 94 |
| yolo11x | detect | ultralytics | 184 | 246 | 5.7× | 4×32 | 1238 MB | 5.2 |
| yoloe-11s-seg | segment (open-vocab) | ultralytics | 177 | 317 | 4.1× | 4×32 | 652 MB | 36 |
| yolo11n-obb | oriented bbox | ultralytics | 175 | 259 | 4.3× | 4×32 | 212 MB | 76 |
| yolo11n-pose | pose | ultralytics | 170 | 269 | 3.7× | 4×32 | 213 MB | 81 |
| dw-ss_ucoco | pose (DWPose) | onnx¹ | 169 | 187 | 1.2× | 1×32 | — | 154 |
| yolo11n-seg | segment | ultralytics | 155 | 243 | 3.6× | 4×32 | 461 MB | 71 |
| 320n | detect² | onnx | 142 | 154 | 1.1× | 4×32 | — | 146 |
| depth_anything_v2_small | depth | transformers | 122 | 132 | 2.1× | 1×32 | 1169 MB | 4.3 |
| yolo11n.onnx | detect | onnx | 102 | 110 | 1.3× | 4×32 | — | 93 |
| dw-mm_ucoco | pose (DWPose) | onnx¹ | 80 | 84 | 1.0× | 1×32 | — | 77 |
| insightface (buffalo_l) | detect (face, SCRFD) | insightface | 70 | 73 | 1.1× | 4×32 | — | 62 |
| dw-ll_ucoco_384 | pose (DWPose) | onnx¹ | 21 | 21 | 1.0× | 1×1 | — | 21 |
| sapiens_0.3b_goliath | body-part seg | torchscript¹ | 3.2 | 3.2 | 1.0× | 1×1 | 2151 MB | timeout |
| yolox_l | detect | onnx³ | 2.6 | 2.6 | 1.0× | 1×1 | — | 2.7 |
| pixai | tagger (multi-label) | onnx | 0.4 | 0.4 | — | 1×1 | — | 0.4 |
| wd-eva02 | tagger (multi-label) | onnx | 0.4 | 0.4 | — | 1×1 | — | 0.4 |
| densepose_r50_fpn | dense UV | torchscript | — | — | — | — | — | — |
| nlf_s / nlf_l | 3D pose | torchscript | — | — | — | — | — | — |

¹ Whole-image nets run one frame per input (no mosaic); tiling doesn't apply — `batch`
is the only lever, so their speedup is modest. ² `320n` loads as a detector but uses a
non-standard 22-channel head (not COCO). ³ `yolox_l` is exported with a **fixed batch
of 1**, so mosaix runs mosaics one-at-a-time — re-export with `dynamic=True` for real
throughput. `densepose`/`nlf` load but expose no generic `forward` (they need their own
project's inference code), so they're recognised but not benchmarkable here.

**Reading it:**
- **Tiling pays off for detectors** — 3–8.5× over untiled, peaking around `grid=4`
  (`4×32`) on this decode-bound 480p clip. RT-DETR and the large/face models gain most.
- **`grid=4` won here, not `grid=9`** — because end-to-end is decode-bound (480p,
  OpenCV); the bigger `9×32` mosaic adds compute without feeding faster frames. On a
  240p source with `decord`, `grid=9`/`16` pull ahead (see performance docs).
- **CPU is viable for nano models** (yolo11n/-pose/-obb, yolov12n-face, 320n: 75–95 FPS)
  but collapses for large ones (RT-DETR, yolo11x, FastSAM-x: 4–5 FPS). Use CPU only for
  the `n`-class models.

### Accuracy & the cost of tiling

Measured on **coco128** (128 labelled COCO images, auto-downloaded, ~7 MB) with one mAP
routine, native full-res vs the default `9×32` mosaic — so the drop *is* the tiling
cost. Packing 9 frames into one 240p-cell mosaic shrinks small objects, so expect a real
hit; it's smallest for big objects / larger models and worst for crowded tiny-object
scenes (use `grid=4` or bigger cells there).

| Model | native mAP@.5 | tiled (9×32) mAP@.5 | retention | published (full COCO / source) |
|---|--:|--:|--:|--|
| rtdetr-l | 0.81 | 0.48 | 59 % | 53.0 AP @.5:.95 |
| yolo11x | 0.71 | 0.41 | 58 % | 54.7 mAP@.5:.95 |
| yolo11m / -seg | 0.66 | 0.38 | 58 % | 51.5 mAP (32.0 mask) |
| yolo11n-seg | 0.48 | 0.23 | 47 % | 32.0 mAP mask |
| yolo11n | 0.46 | 0.21 | 46 % | 39.5 mAP@.5:.95 |
| yolo11n.onnx | 0.46 | 0.19 | 43 % | 39.5 mAP@.5:.95 |

Published metrics for families coco128 can't score (different domains), from each
model's authoritative source:

| Family | Metric (published) |
|---|---|
| YOLO11 pose (n/m/x) | 50.0 / 64.9 / 69.5 mAP-pose @.5:.95 (COCO) |
| YOLO11n-cls | 70.0 % top-1 (ImageNet) |
| YOLO11n-obb | 78.4 mAP@.5 (DOTAv1) |
| Depth-Anything-V2-Small | δ1 ≈ 0.724 (Sun-RGBD) |
| WD-EVA02 tagger | F1 ≈ 0.477 (Danbooru) |
| Sapiens-0.3B Goliath | mIoU 76.7 (body-part seg) |
| InsightFace buffalo_l (SCRFD-10GF) | WIDERFACE 95.2 / 93.9 / 83.1 (easy/med/hard) |
| YOLOX-l | 49.7 AP @.5:.95 (COCO) |

Reproduce everything:

```bash
python benchmarks/bench_all.py --models <models_dir> --video <clip.mp4> --devices cuda,cpu
python benchmarks/accuracy.py  --models <models_dir>     # coco128 native-vs-tiled mAP
```

### Supported model families at a glance

| Family | Examples here | How |
|---|---|---|
| YOLO detect / seg / pose / OBB / cls | yolo11{n,m,x}, `-seg`/`-pose`/`-obb`/`-cls` | `UltralyticsAdapter` (auto) |
| RT-DETR | rtdetr-l | auto (RTDETR) |
| YOLO-World / YOLOE / FastSAM / SAM | yoloe-11s-seg, FastSAM-{s,x} | auto |
| Face detection | yolov12{n,m,l}-face, InsightFace buffalo_{l,s} | auto / `FaceAdapter` |
| Generic YOLO ONNX | yolo11n.onnx, 320n, yolox_l | `OnnxAdapter` (v8/v5 auto) |
| Multi-label taggers | wd-eva02, pixai | `TaggerAdapter` (NCHW/NHWC) |
| Monocular depth (HF) | depth_anything_v2_small | `DepthAdapter` |
| Dense / pose nets (raw fwd) | sapiens, DWPose dw-\* | `TorchScriptAdapter` |
| Loads, needs native API | densepose, nlf | recognised, not run here |
| Anything else | your model | `CallableAdapter` (see below) |

## Plug in *any* vision model

`mosaix` never talks to a model directly — it goes through a thin **adapter**. Three
ways to bring a model, in increasing order of control:

### 1. By file — auto-detected backend

```python
mosaix.VideoPipeline.from_model("yolo11n.pt")        # Ultralytics
mosaix.VideoPipeline.from_model("yolov8n.onnx")      # ONNX Runtime
mosaix.VideoPipeline.from_model("model.engine")      # TensorRT (via Ultralytics)
```

### 2. Any callable — the universal escape hatch

If your model is a Detectron2 net, a Transformers pipeline, a homemade detector —
anything — wrap a function that maps a batch of mosaic images to boxes:

```python
import numpy as np
from mosaix import VideoPipeline, TileConfig, InferenceConfig
from mosaix.adapters import CallableAdapter

def my_detector(mosaics):           # list[BGR uint8] -> list[(N,6)]
    out = []
    for img in mosaics:
        boxes = run_whatever(img)   # (N,6): x1,y1,x2,y2,conf,cls in mosaic pixels
        out.append(np.asarray(boxes, np.float32))
    return out

tile, infer = TileConfig(), InferenceConfig()
adapter = CallableAdapter(my_detector, tile, infer, name="my-model")
pipe = VideoPipeline(adapter)
```

The engine handles all the tiling/batching/remapping; your function only ever sees
ordinary images and returns ordinary boxes. See
[`examples/custom_model.py`](examples/custom_model.py).

### 3. A custom adapter class

Subclass `ModelAdapter`, implement `predict_batch`, and `register_adapter("name", Cls)`
to expose it everywhere. See [`docs/adapters.md`](docs/adapters.md).

---

## Configuration

Everything is a documented dataclass. The defaults are tuned for an 8 GB GPU at 240p.

```python
from mosaix import VideoPipeline, TileConfig, ReaderConfig, InferenceConfig

pipe = VideoPipeline.from_model(
    "yolo11n.pt",
    tile=TileConfig(
        grid=9,            # 9 = 3x3 mosaic; try 4 (2x2) for bigger objects
        batch=32,          # mosaics per forward pass; lower if you OOM
        cell_width=426,    # downscaled frame size
        cell_height=240,
        gutter=12,         # seam padding
    ),
    reader=ReaderConfig(
        stride=1,          # process every frame; 5 = 1-in-5
        threaded=True,     # overlap decode with GPU (essential for real FPS)
        backend="auto",    # "decord" if installed, else "opencv"
    ),
    inference=InferenceConfig(
        device="auto",
        half=True,         # FP16 — ~2x on modern GPUs
        conf=0.25,
        classes=[0],       # keep only COCO 'person'; None = all classes
    ),
)
```

Full reference: [`docs/configuration.md`](docs/configuration.md) ·
Tuning guide: [`docs/performance.md`](docs/performance.md).

---

## Streaming (constant memory on long videos)

`run()` collects every result. For multi-hour videos, `stream()` yields one
`FrameResult` at a time with bounded memory:

```python
pipe = mosaix.VideoPipeline.from_model("yolo11n.pt")
for frame in pipe.stream("8_hour_stream.mp4"):
    process(frame)            # memory stays flat regardless of length
print(pipe.meter.summary())  # FPS, GPU mem, per-stage timing
```

---

## Command line

```bash
mosaix bench yolo11n.pt video.mp4 --grid 9 --batch 32      # measure FPS
mosaix run   yolo11n.pt video.mp4 --out annotated.mp4 --classes 0
mosaix run   yolo11n.pt video.mp4 --jsonl dets.jsonl       # detections to JSONL
mosaix info  video.mp4                                     # probe metadata
```

Every config knob has a flag — run `mosaix bench -h`.

---

## Why "true" throughput

Many benchmarks quote inference-only FPS with decode excluded. `mosaix` reports
**both**: `infer_fps` (GPU forward passes only) and `e2e_fps` (decode + downscale +
mosaic + inference + remap, what you actually get). The threaded reader overlaps
decode with GPU so the end-to-end number stays close to the inference number on long
videos — which is the only number that matters in production.

---

## License

MIT — see [LICENSE](LICENSE).
