Metadata-Version: 2.5
Name: sima-vision
Version: 26.0.4
Summary: Live YOLO computer vision on a SiMa Modalix DevKit 3.0: detection, segmentation and fall detection
Project-URL: Homepage, https://github.com/RizwanMunawar/sima-projects
Project-URL: Repository, https://github.com/RizwanMunawar/sima-projects
Project-URL: Issues, https://github.com/RizwanMunawar/sima-projects/issues
Author: Muhammad Rizwan Munawar
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: computer-vision,devkit,edge-ai,instance-segmentation,modalix,neat,object-detection,palette-sdk,sima,ultralytics,yolo
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: >=3.10
Requires-Dist: pyyaml>=6.0
Provides-Extra: board
Requires-Dist: numpy>=1.24; extra == 'board'
Requires-Dist: opencv-python>=4.7; extra == 'board'
Provides-Extra: compile
Requires-Dist: numpy>=1.24; extra == 'compile'
Requires-Dist: onnx>=1.15; extra == 'compile'
Requires-Dist: onnxsim>=0.4; extra == 'compile'
Requires-Dist: ultralytics>=8.3; extra == 'compile'
Provides-Extra: dev
Requires-Dist: numpy>=1.24; extra == 'dev'
Requires-Dist: opencv-python-headless>=4.7; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

<div align="center">

<img src="assets/sima-devkit-docs-logo-home.png" alt="sima-vision: live YOLO computer vision on a SiMa Modalix DevKit 3.0">
</div>

<div align="center">
<br>

[![SiMa.ai](https://img.shields.io/badge/SiMa.ai-Modalix_DevKit_3.0-E63946)](https://sima.ai)
[![Palette SDK](https://img.shields.io/badge/Palette_SDK-2.1.2-FF8C00)](https://docs.sima.ai)
[![Neat](https://img.shields.io/badge/Neat-0.3.0%20%7C%200.4.0-165432)](https://docs.sima.ai)

[![CI](https://github.com/RizwanMunawar/sima-projects/actions/workflows/ci.yml/badge.svg)](https://github.com/RizwanMunawar/sima-projects/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/badge/pip_install-sima--vision-3775A9&logo=pypi&logoColor=white)](https://pypi.org/project/sima-vision/)
[![Python](https://img.shields.io/badge/python-3.10+-3776AB&logo=python&logoColor=white)](https://www.python.org/)
[![License](https://img.shields.io/badge/license-Apache--2.0-6C757D)](LICENSE)
[![YOLO26](https://img.shields.io/badge/Ultralytics-YOLO26-FFB703&labelColor=333)](https://github.com/ultralytics/ultralytics)
</div>

**Computer vision applications on the SiMa.ai Modalix DevKit 3.0.** Object detection,
instance segmentation, fall detection and retail analytics: one pipeline, no setup step. Needs a
[Modalix DevKit 3.0](https://devkit.sima.ai/products/development-kit-3-0) and Python 3.10
or later.

<div align="center">

| <img src="assets/apps/detect.gif" width="64" alt="Animated object detection icon"> | <img src="assets/apps/segment.gif" width="64" alt="Animated segmentation icon"> | <img src="assets/apps/fall.gif" width="64" alt="Animated fall detection icon"> | <img src="assets/apps/ret-analytics.gif" width="64" alt="Animated shopping trolley icon"> |
|:--:|:--:|:--:|:--:|
| **Object detection** | **Instance segmentation** | **Fall detection** | **Retail analytics** |
| `detect` | `segment` | `fall` | `ret-analytics` |

</div>

- [Quickstart](#quickstart)
- [Apps arguments](#apps-arguments)
- [Your own model](#your-own-model)
- [Moving files](#moving-files)
- [Environment](#environment)

## Quickstart

![on the DevKit](https://img.shields.io/badge/run_on-DevKit-111f68)

Every command here is typed on the board.

```bash
pip install sima-cli

# Login via SiMa.ai Portal
sima-cli login

# Install Neat core, once
sima-cli neat install core@v0.3.0

# Install sima-vision
pip install sima-vision

# Run object detection
sima-vision detect
```

The first run fetches a pretrained YOLO26 pack and a demo clip into `./assets`, then runs.
Every run after that reuses them. Both come from one public GitHub release, so getting
them needs no login and no `sima-cli`.

| App | Fetched for you | Writes |
|:--|:--|:--|
| `detect` | `yolo26n-det-bf16-mla_tess-b1.tar.gz` (21 MB) + a 1080p demo clip (13 MB) | `detections.mp4` |
| `segment` | `yolo26n-seg-bf16-mla_tess.tar.gz` (24 MB) + the same clip | `segmentation.mp4` |
| `fall` | the detection pack again + a shorter clip (1.2 MB) | `falls.mp4` |
| `ret-analytics` | the detection pack + `shopping-trolley-items.mp4` | `retail.mp4` |

Each app writes annotated video. Stills are off unless you ask for them
with `--save`.

<details>
<summary>&nbsp;<b>Instance segmentation</b> &nbsp;·&nbsp; per-pixel masks, with the background blurred</summary>

<br>

```bash
sima-vision segment                        # background blurred, instances sharp
sima-vision segment --keep-classes person  # only people stay sharp
sima-vision segment --anonymise            # the other way round: blur the people
sima-vision segment --no-blur              # a plain overlay, nothing blurred
```

</details>

<details>
<summary>&nbsp;<b>Fall detection</b> &nbsp;·&nbsp; tracks people and relabels the box when one goes down</summary>

<br>

```bash
sima-vision fall
sima-vision fall --classes person --confirm 2.0
```

The frame is the one `detect` draws: same palette, same boxes, same captions. A confirmed
fall changes one thing, the class the box is labelled with, so a person reading `person
0.93` a second earlier now reads `FALL 0.93`. Each fall is also printed on the console and
counted in the run summary.

A confirmed fall re-arms only after sustained recovery.

</details>

<details>
<summary>&nbsp;<b>Retail analytics</b> &nbsp;&middot;&nbsp; displays items in a shopping trolley</summary>

<br>

```bash
sima-vision ret-analytics
sima-vision ret-analytics --source shopping-trolley-items.mp4
sima-vision ret-analytics --classes bottle banana apple
```

Displays detected trolley items with boxes, names and confidence, and writes
`retail.mp4`. Uses the same source, model and recording flags as `detect`.
The white side menu uses a table with Qty, Item, Unit $, and Total $ columns,
larger text, alternating rows, and an estimated basket total.
Each product row is bold and has its own consistent color.
Products enter the menu after three consecutive detections and stay in first-added
order. Quantities increase only after a higher simultaneous count is confirmed for
three frames. Brief misses never remove rows or reduce counts, and repeated frames
never add purchases. The menu retains confirmed items for the run; removals are not
inferred, and new items hidden behind existing ones cannot be counted reliably.
All model classes are displayed by default; `--classes` optionally filters them.
The default sample is `shopping-trolley-items.mp4`, fetched automatically.
The default model recognizes COCO categories such as bottle, banana, apple,
and orange. Product-specific recognition needs a trained model and matching labels.
Point the camera at the trolley so the displayed detections represent its contents.

The optional `retail.classes` config setting selects item names or ids;
an empty list displays all model classes.

Prices are rough per-item US estimates, not live quotes. Produce estimates use
[US retail listings](https://www.walmart.com/cp/fresh-produce/976793) as a reference.
Unpriced classes show `price N/A` and are excluded from the known subtotal.
Defaults include cherry ($0.10 each), cantaloupe ($3.00 each), banana ($0.20 each),
cucumber ($0.75 each), bread ($2.00 per loaf), snack ($1.50 per small packet),
orange ($1.00 each), avocados ($1.00 each), egg ($0.25 each), and milk
($4.00 per gallon container). Both `avocado` and `avocados` labels are supported.
Override estimates for your products in `config.yaml`:

```yaml
retail:
  classes: []
  prices:
    banana: 0.20
    tomato: 0.25
    apple: 1.00
```

The COCO model does not have a tomato class; tomato detection requires a model
trained for tomatoes and a matching labels file.

If an automatically discovered `config.yaml` contains fall settings without a
`retail` section, retail uses its own sample and output paths and prints a warning.
CLI flags still win; an explicit `--config` uses the named file as written.

</details>

## Apps arguments

Every flag, and which apps take it. All of them are for the four apps, which run on the
DevKit. `sima-vision <app> --help` prints the same list.

| Flag | Apps | What it does |
|:--|:--|:--|
| `--source`, `-s URI` | all | File, `https` URL, RTSP URL, or empty for the sample clip |
| `--source-type` | all | `video`, `rtsp` or `usb`. Default `video` |
| `--fps N` | all | Source frame rate. Default `0`, read from the stream |
| `--width PX` | all | Source width. Default `0`, read from the stream's SPS |
| `--height PX` | all | Source height. Default `0`, read from the stream's SPS |
| `--model`, `-m PATH` | all | Compiled pack, or an `https` URL to one. Empty fetches this app's default |
| `--labels PATH` | all | Newline-separated class names. Defaults to the packaged COCO list |
| `--family NAME` | all | Detection head. Must match the model or every box is noise |
| `--conf T` | all | Minimum confidence. Default `0.30` |
| `--iou T` | all | Non-max suppression IoU. Default `0.60` |
| `--max-det N` | all | Top-K cap per frame. Default `50` |
| `--frames`, `-n N` | all | Stop after N frames. Default `0`, runs until interrupted |
| `--timeout MS` | all | How long to wait for a frame before giving up. Default `20000` |
| `--video-path PATH` | all | Where the annotated recording is written |
| `--no-video` | all | Do not record |
| `--video-encoder` | all | `sima` (hardware H.264, default) or `opencv` (software, about ten times slower) |
| `--video-bitrate KBPS` | all | Target bitrate for the hardware encoder. Default `12000` |
| `--save` | all | Also write annotated stills. Off by default, so only the video is written |
| `--save-dir DIR` | all | Where annotated stills are written. Implies `--save` |
| `--save-every N` | all | Write every Nth still. Default `10` once stills are on; implies `--save`. `0` disables |
| `--no-save` | all | Do not write stills. Already the default; use it to override a config file |
| `--show-fps` | all | Show the FPS badge on the overlay (hidden by default) |
| `--no-hud` | all | Leave the frame-rate badge off the overlay |
| `--hud-scale N` | all | Frame-rate badge font size. Default `2.4`, which is 1.5x the caption scale; `0` follows the caption scale exactly |
| `--hud-thickness N` | all | Badge stroke weight. Default `6`, which is 1.5x the caption thickness; `0` follows it exactly |
| `--hud-bg B,G,R` | all | Badge fill colour. Default `132,28,193` (`#C11C84`) |
| `--hud-color B,G,R` | all | Badge text colour. Default `255,255,255` |
| `--hud-padding PX` | all | Gap between badge text and its edge, which is what sizes the badge. Default `22`; `0` follows the caption padding |
| `--config`, `-c PATH` | all | Config file. Defaults to `./config.yaml` |
| `--no-config` | all | Ignore any config file and use defaults plus these flags |
| `--validate` | all | Resolve and check the settings, then stop. Needs no board |
| `--quiet`, `-q` | all | Warnings, errors and the closing report only |
| `--profile` | all | Per-stage timings, when a run is slower than it should be |
| `--queue-depth N` | all | Neat's own queue depth. Below `4` the graph drops frames when the recorder falls behind. Default `4` |
| `--sink-queue-depth N` | all | Finished frames that may wait for the recorder. Host memory only. Default `12` |
| `--sink-queue-mb MB` | all | Memory budget for that backlog, which grows it to fit a known clip |
| `--output-buffers N` | all | Buffers each public output may hold. Default `1` |
| `--decoder-buffers N` | all | Buffers to ask the decoder for. Default `0`, sized from the stream's reference frames |
| `--decoder-tuning` | all | Hardware decoder preset. Default `default`, which keeps every frame; `auto` drops frames in bursts |
| `--segment-frames N` | all | Frames per piece when a clip is too long for one decode. Default `150`; `0` runs it whole |
| `--blur` / `--no-blur` | `segment` | Blur the background and keep instances sharp. On by default; `--no-blur` draws a plain overlay |
| `--blur-method` | `segment` | `gaussian`, `pixelate` or `none`. Default `gaussian` |
| `--blur-strength PX` | `segment` | Gaussian kernel width at 1080p. Default `41` |
| `--keep-classes CLASS...` | `segment` | Names or ids that stay sharp. Default: every detected class |
| `--anonymise`, `--anonymize` | `segment` | Blur the instances instead of the background |
| `--mask-threshold T` | `segment` | Mask cut-off as a probability. Default `0.5`; lower grows instances |
| `--no-masks` | `segment` | Blur around plain boxes instead. Needs no segment head |
| `--minimal` | `segment` | Pull frames and do nothing else. Tells a slow app from a stalled graph |
| `--classes CLASS...` | `fall`, `ret-analytics` | Fall classes (default `person`) or retail products (default all model classes) |
| `--confirm S` | `fall` | How long a fall signal must hold before it counts. Default `1.5` |
| `--no-fall` | `fall` | Track without judging falls, which is how you tune tracking first |

A `config.yaml` in the working directory is picked up on its own. Flags beat it, and it
beats the built-in defaults.

## Your own model

![on your PC](https://img.shields.io/badge/run_on-Host_PC-64239E)

A trained `.pt` has to be compiled into a `.tar.gz` pack before the board can run it. The
compiler is the `afe` package and exists only inside the Palette Model SDK container,
which is x86 Docker, which on Windows means WSL2. So the whole of this runs on your PC,
needs a [community.sima.ai](https://community.sima.ai) account and ~10 GB of disk, and
takes about 30 minutes the first time. [Quickstart](#quickstart) needs none of it.

<details>
<summary>&nbsp;<b>Converting a trained <code>.pt</code> into a DevKit pack</b></summary>

<br>

```bash
# --- PowerShell, as Administrator -------------------------------------------
wsl --install -d Ubuntu
wsl -l -v                             # want: Ubuntu, Running, 2

# Install Docker Desktop, then Settings > Resources > WSL integration > Ubuntu.
# Without that toggle docker works in PowerShell and is missing inside Ubuntu.

wsl -d Ubuntu                         # everything below is Ubuntu. exit comes back

# --- Ubuntu ------------------------------------------------------------------
docker run hello-world                # must print "Hello from Docker!"

sudo apt update && sudo apt install -y python3-venv python3-pip
python3 -m venv ~/sima                # Ubuntu refuses a bare pip install
source ~/sima/bin/activate            # new shells start outside it. Look for (sima)
pip install sima-cli
sima-cli login
sima-cli install ghcr:sima-neat/sdk:v2.0.0    # several GB, once per machine

cd /path/to/your/model                # the folder holding best.pt
sima-cli sdk setup --workspace .      # . is what gets mounted. 15-20 minutes
activate-model-compiler

# --- Model SDK shell ---------------------------------------------------------
pip install "sima-vision[compile]"    # ultralytics, onnx, onnxsim
sima-vision compile best.pt           # ONNX, bfloat16, tessellate, ELF. 10-15 minutes
                                      # -> build/best_mpk.tar.gz

# --- on the DevKit -----------------------------------------------------------
sima-vision push build/best_mpk.tar.gz
sima-vision detect --model best_mpk.tar.gz
sima-vision detect --model https://example.com/my-model.tar.gz   # a URL works too
```

</details>

## Moving files

![on your PC](https://img.shields.io/badge/run_on-Host_PC-64239E)

Output lands beside the run, on the board, and both commands are typed on your PC. Name the board once and neither command needs
`--host`:

```powershell
$env:SIMA_VISION_DEVKIT="sima@<devkit-ip>"
```

```bash
sima-vision pull                     # everything the run left
sima-vision pull --into results/
sima-vision push my-clip.h264
```

The default pull includes `retail.mp4` and `retail-report.json` when present,
along with the other apps' outputs. Missing files are skipped.

## Environment

| Variable | What it does |
|:--|:--|
| `SIMA_VISION_DEVKIT` | The board, as `user@address`, so `push` and `pull` stop asking |
| `SIMA_VISION_ASSETS` | Where clips and models are downloaded. Default `./assets` |
| `SIMA_VISION_PYNEAT` | The `pyneat` virtualenv, when the search does not find it |
| `SIMA_VISION_PYNEAT_INDEX` | A pip index carrying a `pyneat` wheel, if your site publishes one |
| `SIMA_VISION_AUTO_INSTALL` | `0` to look but never install |
| `SIMA_VISION_QUIET` | Non-empty is `--quiet` for every command |
| `SIMA_VISION_COLOR` | `0` or `1` to force colour off or on. `NO_COLOR` also works |

## Contributing

```bash
git clone https://github.com/RizwanMunawar/sima-projects.git
cd sima-projects
pip install -e ".[dev]"

ruff check sima_vision tests
pytest -q
```

The tests need no board.

## License

The models used here for testing are **Ultralytics YOLO26**, under **AGPL-3.0**. All other
parts of this repository are under **Apache-2.0**. See [LICENSE](LICENSE).

## Credits

- [SiMa.ai](https://github.com/SiMa-ai) for Modalix, the Palette SDK and Neat
- [Ultralytics](https://github.com/ultralytics/ultralytics) for the YOLO26 models
- [Timothy Do](https://github.com/dotimothy) for the Neat 0.4.0 port, frame-complete
  recordings and hardware H.264 recording

<div align="center">

Built by **Muhammad Rizwan Munawar**. If this saved you an afternoon, **star the repo**
and pass it on to someone else bringing up a DevKit.

<a href="https://github.com/RizwanMunawar"><img src="assets/socials/github.svg" width="50" alt="GitHub"></a>
&nbsp;&nbsp;
<a href="https://www.linkedin.com/in/muhammadrizwanmunawar/"><img src="assets/socials/linkedin.svg" width="50" alt="LinkedIn"></a>
&nbsp;&nbsp;
<a href="https://x.com/muhammdrizwanmr"><img src="assets/socials/x.svg" width="50" alt="X"></a>
&nbsp;&nbsp;
<a href="https://www.youtube.com/@muhammadrizwanmunawar"><img src="assets/socials/youtube.svg" width="50" alt="YouTube"></a>
&nbsp;&nbsp;
<a href="https://muhammadrizwanmunawar.medium.com/"><img src="assets/socials/medium.svg" width="50" alt="Medium"></a>

</div>
