Metadata-Version: 2.4
Name: trio-retina
Version: 0.2.1
Summary: Model-agnostic computer-vision pipeline: turn any detector (YOLO, VLM, Grounding DINO) into structured tracking events — zone intrusion, line-crossing, dwell, counting — over video or RTSP. The state layer for world models.
Project-URL: Homepage, https://github.com/machinefi/trio-retina
Project-URL: Source, https://github.com/machinefi/trio-retina
Project-URL: Documentation, https://github.com/machinefi/trio-retina#readme
Project-URL: Issues, https://github.com/machinefi/trio-retina/issues
Project-URL: Changelog, https://github.com/machinefi/trio-retina/blob/main/CHANGELOG.md
Author: MachineFi Labs
License: Apache-2.0
License-File: LICENSE
Keywords: computer-vision,digital-twin,edge-ai,event-driven,grounding-dino,line-crossing,object-detection,object-tracking,opencv,people-counting,perception,rtsp,ultralytics,video-analytics,vision-pipeline,vlm,world-model,yolo,zone-intrusion
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: numpy>=1.23
Provides-Extra: all
Requires-Dist: norfair>=2.2; extra == 'all'
Requires-Dist: opencv-python>=4.8; extra == 'all'
Requires-Dist: pillow>=10; extra == 'all'
Requires-Dist: torch>=2.0; extra == 'all'
Requires-Dist: transformers>=4.40; extra == 'all'
Requires-Dist: ultralytics>=8.3; extra == 'all'
Provides-Extra: dev
Requires-Dist: norfair>=2.2; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.26; extra == 'docs'
Provides-Extra: grounding
Requires-Dist: pillow>=10; extra == 'grounding'
Requires-Dist: torch>=2.0; extra == 'grounding'
Requires-Dist: transformers>=4.40; extra == 'grounding'
Provides-Extra: norfair
Requires-Dist: norfair>=2.2; extra == 'norfair'
Provides-Extra: video
Requires-Dist: opencv-python>=4.8; extra == 'video'
Provides-Extra: yolo
Requires-Dist: ultralytics>=8.3; extra == 'yolo'
Description-Content-Type: text/markdown

# Trio Retina

**Turn any perception model's output into one standard, queryable world-state — symbolic events, with a latent-vector channel built in.**
The model-agnostic state layer for world models.

*A lightweight, model-agnostic **computer-vision pipeline** for **object detection & tracking** that emits structured **events** — zone intrusion, line-crossing, dwell, people-counting — from **YOLO**, **VLM**, or **Grounding DINO** detectors over video, files, or **RTSP**. Runs on CPU at the **edge**; feeds **digital twins**, dynamics models, and LLMs.*

> Just want camera events (zone intrusion, line-crossing) pushed to a webhook? → jump to the [5-line quickstart](#-quickstart), or copy [`examples/rtsp_to_webhook.py`](examples/rtsp_to_webhook.py).

[![CI](https://github.com/machinefi/trio-retina/actions/workflows/ci.yml/badge.svg)](https://github.com/machinefi/trio-retina/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/trio-retina.svg)](https://pypi.org/project/trio-retina/)
[![Docs](https://img.shields.io/badge/docs-live-brightgreen.svg)](https://machinefi.github.io/trio-retina/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)

![Trio Retina computer-vision pipeline: YOLO object tracking with two dynamics models forecasting entity trajectories from one world-state](https://raw.githubusercontent.com/machinefi/trio-retina/main/media/retina_demo.gif)

> One world-state from any detector → **two dynamics models forecast where each entity is headed** off the *same* state (gray = constant-velocity baseline, magenta = a learned model). Swap the detector (YOLO → V-JEPA → DINO) or the dynamics model — the state in the middle is the constant.

## 👋 hello

**Trio Retina** (Retina for short) turns raw signals — video, sensor — into a **queryable world-state**: readable **events** (`zone.enter`, `dwell`, `line.cross`) *plus* a standardized **latent** `vec` channel on the same records, on one small model-agnostic standard. The latent channel is a real, serializable interface today (attach your own embedding — see [`examples/latent_vec.py`](examples/latent_vec.py)); the automatic *producers* (V-JEPA scene + per-object ReID) are on the [roadmap](#-roadmap). Bring any model (YOLO, V-JEPA, DINO, a VLM, or none); Retina assembles its output into state a dynamics model, rule engine, or LLM can consume.

Think **OpenTelemetry for perception** — it doesn't build the sensors, it normalizes any of them into one state. In world-model terms it's the **encoder** (`s = Enc(x)`), and *only* the encoder; dynamics and policy build on top. → see [`DESIGN.md`](DESIGN.md).

## 💻 install

```bash
pip install trio-retina            # core: numpy only
pip install 'trio-retina[yolo]'    # + Ultralytics YOLO adapter
pip install 'trio-retina[video]'   # + OpenCV frame source (files / RTSP / webcam)
pip install 'trio-retina[all]'     # everything
```

## 🔥 quickstart

Runs on a bare `pip install trio-retina` (numpy only) — no model, no GPU, no video file. A stand-in detector walks one "person" across a dock zone; Retina emits the real `retina.event` stream:

```python
import numpy as np

from retina import CountRule, IoUTracker, Retina, Zone, ZoneRule
from retina.detect import Detection


class ScriptedDetector:
    """A stand-in model: one 'person' walking across a dock zone."""

    def __init__(self):
        self._xs = list(range(0, 102, 6))

    def __call__(self, frame):
        x = self._xs.pop(0) if self._xs else 100
        return [Detection(label="person", bbox=(x - 10, 40, x + 10, 60), confidence=0.9)]


dock = Zone("dock", [(40, 0), (60, 0), (60, 100), (40, 100)])

cam = Retina(
    source_id="cam_01",
    detector=ScriptedDetector(),
    tracker=IoUTracker(min_hits=2),
    rules=[
        ZoneRule(dock, classes={"person"}, dwell_s=2.0),
        CountRule(1, classes={"person"}),
    ],
)

frames = [(np.zeros((100, 100, 3), dtype=np.uint8), float(i)) for i in range(18)]
for event in cam.run(frames):
    print(event.to_json())
    # {"type":"count.threshold","t":1.0,"src":"cam_01","n":1,"frame":1,...}
    # {"type":"zone.enter","t":7.0,"src":"cam_01","id":1,"label":"person",...}
    # {"type":"zone.dwell","t":7.0,...,"zone":"dock","dur":2.0,...}
    # {"type":"zone.exit","t":7.0,...,"zone":"dock","dur":3.0,...}
```

**▶ with a real model + video** — `pip install 'trio-retina[yolo]'` (add `[video]` for the frame source), then point it at your clip:

```python
from retina import Retina, Zone, ZoneRule, YoloDetector
from retina.sources import video_frames

dock = Zone("dock", [(0.3, 0.2), (0.7, 0.2), (0.7, 0.9), (0.3, 0.9)], normalized=True)

cam = Retina(
    source_id="cam_01",
    detector=YoloDetector("yolo11n.pt", classes={"person"}),
    rules=[ZoneRule(dock, classes={"person"}, dwell_s=30)],
)
for event in cam.run(video_frames("your.mp4")):
    print(event.to_json())
    # {"type":"zone.dwell","t":1718254799.8,"src":"cam_01","id":42,
    #  "label":"person","zone":"dock","dur":31.0,"conf":0.91}
```

More no-model examples ship with the source (not the wheel) — `git clone` the repo and run `python examples/quickstart.py` (the forecast / video demos need `[video]` + a clip).

**▶️ Or run it in your browser — no install:**

| notebook | what it shows |
|---|---|
| [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/machinefi/trio-retina/blob/main/notebooks/01_quickstart_events.ipynb) | **quickstart** — detector → `zone` / `line` / `count` / `dwell` events + `validate()` |
| [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/machinefi/trio-retina/blob/main/notebooks/02_camera_to_webhook.ipynb) | **camera → webhook** — a restricted-zone alert pushed to your endpoint |
| [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/machinefi/trio-retina/blob/main/notebooks/03_from_supervision.ipynb) | **from Supervision** — pipe your existing `sv.Detections` straight in |

### compose models with `|`

Wire models like n8n / LangChain, no GUI. Add a cheap gate and a VLM enricher anywhere in the chain:

```python
from retina import MotionGate, GateNode, YoloDetector, IoUTracker, EnricherNode, ZoneRule, JsonlSink

pipe = (
    GateNode(MotionGate())                 # skip static frames (cut model calls)
    | YoloDetector("yolo11n.pt", classes={"person", "forklift"})
    | IoUTracker()
    | EnricherNode(my_vlm_describe)        # attach a VLM read to frame.user
    | ZoneRule(dock, dwell_s=30)
    | JsonlSink("events.jsonl")
)
```

<details>
<summary>Two more ways to wire it (explicit list · declarative JSON) + the node catalog</summary>

```python
# explicit node list
from retina import Pipeline, DetectorNode, TrackerNode, RuleNode
pipe = Pipeline([DetectorNode(yolo), TrackerNode(), RuleNode(ZoneRule(dock))])

# declarative workflow file (shareable, no code)
pipe = Pipeline.from_json("workflow.json")   # see examples/workflow.json
```

| node | what it does | wraps |
|---|---|---|
| `DetectorNode` | image → detections | any `callable(image)->[Detection]` |
| `TrackerNode` | detections → tracks | `IoUTracker` / `NorfairTracker` |
| `RuleNode` | tracks → events | `ZoneRule` / `LineRule` / `CountRule` |
| `GateNode` | drop uninteresting frames | any `callable(image,t)->bool` (e.g. `MotionGate`) |
| `EnricherNode` | attach context to `frame.user` | any `callable(frame)->dict` (VLM / V-JEPA) |
| `SinkNode` | emit events | `JsonlSink` / `WebhookSink` |

Register your own for `from_json` with `register_node("my_type", builder)`.
</details>

## 🎛️ supported models

Retina imports no model — **any** detector plugs in, and out comes one standard event stream. That seam *is* the point:

| plug in any detector… | → | …out comes one `retina.event` stream |
|---|:---:|---|
| **YOLO** (Ultralytics: v5–v12, RT-DETR) | → | `{"type":"zone.enter", "id":42, "label":"person", …}` |
| **any VLM** (GPT-4o · Qwen-VL · Gemini · Claude) | → | `{"type":"line.cross", "dir":"a_to_b", …}` |
| **Grounding DINO** (open-vocab, no training) | → | `{"type":"zone.dwell", "dur":31.0, …}` |
| your existing **`sv.Detections`** (Supervision) | → | `{"type":"count.threshold", "n":12, …}` |
| any **`callable(image) -> [Detection]`** | → | …+ an optional latent `vec` on the same record |

Supervision gives you boxes on a screen; Retina turns *any* of those into a serializable state + event stream the next layer (dynamics, twin, agent) can consume. Batteries-included adapters:

- **YOLO family** — `YoloDetector("<weights>.pt")` (Ultralytics): YOLOv5/8/9/10/11/12, RT-DETR. Open-vocab via YOLO-World.
- **Open-vocab from text** — `GroundingDinoDetector(["forklift", "hard hat"])`, no training.
- **Any VLM** — `VlmDetector(client, prompt)` (Qwen-VL / Gemini / GPT-4o / Claude / local), as a detector or an event-source enricher.
- **Supervision interop** — `Detection.from_supervision(sv_detections)` ingests a Roboflow `sv.Detections`, so anything that already converts to Supervision pipes straight into Retina's event layer.

Trackers are pluggable too: `IoUTracker` (pure-Python default) or `NorfairTracker`.

## 📦 the event format

The `retina.event` standard is tiny, like a JWT — three required fields, everything else optional and omitted when absent. Full spec in [`SPEC.md`](SPEC.md).

```json
{"type":"zone.dwell","t":1718254799.8,"src":"cam_01","id":42,"label":"person","zone":"dock","dur":31.0}
```

```python
from retina import validate
validate(event)   # -> [] if valid, else a list of problems  (pure-Python, ships a JSON Schema)
```

## 🎬 demos

### Forecast — the dynamics layer on top of Retina

The hero GIF above. [`examples/forecast/`](examples/forecast/) runs a dynamics model on Retina's `WorldState` stream and shows *why* Retina is necessary: a dynamics model eats structured **state**, not pixels.

### iTwin.js — a live, predictive layer for a digital twin

![Trio Retina perception events and forecast arrows rendered live on a Bentley iTwin.js digital twin (Baytown plant)](https://raw.githubusercontent.com/machinefi/trio-retina/main/examples/itwin/media/retina_itwin_demo.gif)

[`examples/itwin/`](examples/itwin/) drops Retina's entities, forecast arrows, and `retina.event` alerts onto a real Bentley **iTwin.js** iModel (the Baytown sample plant), through one neutral JSON contract — rendered fully headless. Retina doesn't replace the twin; it gives it *live eyes*.

<details>
<summary>All examples</summary>

The examples live in this repo (not in the installed wheel) — `git clone` to run them. The top-level quickstarts run with **no model and no GPU** (synthetic detections):

```bash
python examples/quickstart.py          # zone / line / count / dwell events
python examples/three_apps.py          # one stream -> security, retail, safety
python examples/any_model.py           # swap the detector, rest unchanged
python examples/gate_savings.py        # a cheap gate cuts detector calls 100 -> 23
python examples/pipeline_compose.py    # compose with | (n8n without a GUI)
python examples/rtsp_to_webhook.py     # camera -> restricted-zone alert -> webhook
python examples/from_supervision.py    # ingest a Roboflow sv.Detections pipeline
python examples/latent_vec.py          # populate the latent vec channel by hand
```

Real-footage / dynamics demos need a clip and the extras — `pip install 'trio-retina[all]'`:

```bash
python examples/yolo_video.py v.mp4    # YOLO on a video file
examples/forecast/                     # dynamics layer on the WorldState stream (needs [video] + a clip)
examples/itwin/                        # events + forecasts on a Bentley iTwin.js iModel
```
</details>

**Send events anywhere.** `WebhookSink(url)` POSTs each event as JSON (stdlib urllib, no `requests`); `JsonlSink(path)` streams to a file. For a live camera, `video_frames(src, live=True)` reads RTSP / HLS / webcam with wall-clock timestamps — see [`examples/rtsp_to_webhook.py`](examples/rtsp_to_webhook.py).

## 🎯 use cases

One state layer, many domains — the *same* `retina.event` stream, read differently above:

- **Security & intrusion detection** — `zone.enter` / `line.cross` on cameras and RTSP feeds.
- **Retail analytics & people-counting** — footfall, queue dwell, zone occupancy from any detector.
- **Workplace safety** — PPE, forklift, and restricted-zone alerts via open-vocab detectors.
- **Smart city & traffic monitoring** — vehicle/pedestrian counting and crossings at the edge.
- **Industrial digital twins** — feed live entities + forecasts into a twin ([iTwin.js demo](examples/itwin/)).

## 🧠 how it works

Everything flows through one append-only data unit, the **`Frame`**. Each stage *enriches* it and never overwrites upstream fields:

```
                      ┌──────────────── Frame (append-only) ───────────────┐
 frame ─► Detector ─► │ .detections ─► Tracker ─► .tracks ─► Rule ─► .events │ ─► Sink
   ▲        ▲         │                  ▲                    ▲              │     ▲
 source   any model   │   Gate (skip?)   tracker     zone/line/count/dwell  │  jsonl/
                      │   Enricher (VLM / V-JEPA → .user)                    │  webhook
                      └─────────────────────────────────────────────────────┘
```

- The **detector** is the model-agnostic seam: any `callable(image) -> [Detection]`.
- The **tracker** gives objects identity over time; **rules** turn tracks into **events**; **enrichers** attach context; **gates** skip work; **sinks** push out.
- Output is **dual**: a readable symbolic stream *and* an optional model-tagged latent channel — never collapsed.

<details>
<summary>Why "encoder", the dual state, and how it compares to DeepStream / Supervision</summary>

**Two senses of "encoder."** Foundation backbones (V-JEPA, DINO, SAM, YOLO) turn pixels into features — that race is theirs, and Retina rides it. Retina is the encoder *layer* on top: it **fuses** many models into one record, gives objects **persistent identity**, **structures** it into entities + relations + events, carries the **dual** symbolic + latent channels, as an **event-sourced stream** — one small, serializable, model-agnostic standard.

**Dual state.** The same entities on two linked channels: *symbolic* (readable `events` / entity records, for rules / LLMs / dashboards) and *latent* (optional model-tagged embeddings, for a downstream dynamics model). Symbols you can read; vectors a model can predict on. The latent channel is a standardized, serializable interface shipping today — you can populate `entity.vec` with your own embedding now ([`examples/latent_vec.py`](examples/latent_vec.py)); the built-in *producers* (V-JEPA / ReID) are on the roadmap, not shipped yet.

**vs DeepStream / Holoscan** — same good ideas (event semantics, metadata model, composable graph), none of the weight:

| | DeepStream / Holoscan | **Retina** |
|---|---|---|
| Install | CUDA + TensorRT + containers | `pip install trio-retina` |
| Hardware | NVIDIA / Jetson locked | any machine — CPU is fine |
| Model | tied to the NV stack | **bring any model** (or none) |
| Shape | a platform you build *inside* | a library you `import` |
| Core deps | a lot | **numpy** |

**vs Supervision** — Supervision turns a model's output into detections + overlays (great toolbox, ends at the screen). Retina is a level up: it emits a serializable **state + event stream** that the *next* layer (dynamics, twin, agent) consumes. We compose Supervision / detectors, not compete with them.

Full rationale, references, and the world-model stack: [`DESIGN.md`](DESIGN.md).
</details>

## 🗺️ roadmap

Early but real (`v0.2.1`). Stable: the event layer + JSON Schema/validator, the composable pipeline (`|` / list / JSON), YOLO + open-vocab + VLM detectors (plus `from_supervision` interop), IoU + Norfair trackers, and jitter-robust rules (`exit_grace_s` · `anchor` · `min_frames`).

Next: ByteTrack / OC-SORT · `proximity` / `anomaly` events · VLM-as-event-source · Kafka / MQTT sinks · the **latent channel** (surface V-JEPA scene + per-object embeddings). See [`CHANGELOG.md`](CHANGELOG.md).

Retina is the open **perception encoder** extracted from [Trio](https://machinefi.com); the layers above (dynamics, policy / judgment) are Trio's commercial platform. Retina is, and stays, model-agnostic and free.

## 🤝 contributing

Contributions that keep Retina small and beautiful are very welcome — see [`CONTRIBUTING.md`](CONTRIBUTING.md) for dev setup and how to add a detector / tracker / rule / sink. By participating you agree to the [Code of Conduct](CODE_OF_CONDUCT.md); to report a vulnerability see [`SECURITY.md`](SECURITY.md).

## license

[Apache-2.0](LICENSE).
