Metadata-Version: 2.5
Name: toposync-ext-vision
Version: 0.9.0
Summary: Toposync first-party extension: task-oriented vision operators.
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: cryptography<47,>=44
Requires-Dist: httpx<1,>=0.27
Requires-Dist: huggingface-hub<1,>=0.34
Requires-Dist: norfair<3,>=2.2
Requires-Dist: numpy<3,>=1.26
Requires-Dist: onnx<2,>=1.17
Requires-Dist: onnxruntime<2,>=1.30
Requires-Dist: pillow<13,>=10
Requires-Dist: safetensors<1,>=0.4
Requires-Dist: tokenizers<1,>=0.22
Requires-Dist: toposync-core>=0.9.0
Requires-Dist: toposync-ext-cameras>=0.1.0
Description-Content-Type: text/markdown

# Toposync Vision extension

First-party extension focused on public task-oriented vision operators for the Pipelines runtime.

## What it provides

- `vision.detect`
- `vision.track`
- `vision.group_events`
- `vision.spatial_relation_event` confirms proximity after observed dwell. A valid matched pair stays stable when another member moves closer. Replacing either participant closes the previous occurrence and starts a new dwell; time without a new observation cannot confirm a candidate. These events describe proximity, not intent or recognition across separate visits.
- `vision.identity_evidence` and `vision.recognize_identity` (explicit activation; [setup, curation and recovery](docs/IDENTITY_RECOGNITION.md))
- `vision.crop_objects`
- `vision.segment_instances`
- `vision.pose_estimate` (experimental MediaPipe Pose 33 and RTMPose-M Halpe26 backends)
- `vision.gesture_recognize` (experimental temporal body gestures)

The public surface is task-based, not vendor-based. The official first-party runtime is ONNX Runtime, with CPU as the default execution path.

For spatial relations, `dwell_seconds` accumulates intervals between consecutive positive observations. Loss of evidence pauses this duration; recovery preserves the occurrence within its grace period without adding the missing interval. Closing grace and source timeouts never extend the observed duration. This remains an estimate from observation cadence, not proof of continuous physical presence between frames.

Spatial relation state is isolated by source, camera and group. Replayed terminal
groups cannot open another occurrence. Group producers attach a runtime epoch and
monotonic lifecycle sequence, allowing the shared terminal-subject ledger to compact
closed groups without dropping replay protection after 4,096 occurrences. A new
producer epoch can reuse a closed identifier; this does not establish person identity
across sessions or qualify recovery when the previous group never closed.

Proximity observations require valid, recent capture evidence. Dwell follows media
time rather than processing delay; same-frame member updates may update geometry
but cannot add elapsed time. Older capture generations/sequences are rejected.
A capture reset closes the old relation and starts fresh confirmation; a late close
from the old generation cannot terminate the new relation. Publication freshness
does not prove physical camera latency or synchronized remote clocks.

## Dependencies

- The default `toposync` application bundle includes the first-party ONNX Runtime CPU stack.
- The GPU-oriented first-party bundles are published separately:
  - `toposync-vision-cuda`
  - `toposync-vision-directml`
- The official extension package now also includes the first-party tracker stack:
  - `byte_world`
  - `simple_iou_kalman`
  - `norfair`

## Notes

- `vision.detect` resolves `ModelManifest` entries and builds an ONNX Runtime backend automatically when `runtime=onnxruntime`.
- By default, the ONNX Runtime backend prefers `CPUExecutionProvider`. Optional acceleration is opt-in via the `toposync-vision-cuda` / `toposync-vision-directml` bundles or `TOPOSYNC_VISION_ONNXRUNTIME_PROVIDERS`.
- Manifest files can be loaded from `TOPOSYNC_VISION_MANIFESTS_DIR` or `TOPOSYNC_VISION_MANIFEST_PATHS`.
- Custom manifests imported from the UI are persisted under `.toposync-data/vision-manifests/` for the selected processing server.
- Built-in first-party manifests ship inside the wheel, but their ONNX weights do not. When no checkout-local artifact exists, official model ids resolve to `TOPOSYNC_DATA_DIR/vision-models/...` (or `.toposync-data/vision-models/...` by default).
- The extension now ships a built-in RTMDet detection shortlist in `extensions/vision/manifests/`:
  - `rtmdet_det_tiny`
  - `rtmdet_det_small`
  - `rtmdet_det_medium`
- The extension now also ships a built-in RF-DETR detection shortlist in `extensions/vision/manifests/`:
  - `rfdetr_det_nano`
  - `rfdetr_det_small`
  - `rfdetr_det_medium`
- In a source checkout, existing local artifacts under `extensions/vision/models/...` are still honored.
- In installed environments, official artifacts belong under the managed model store in `TOPOSYNC_DATA_DIR/vision-models/...` and are intentionally not bundled in the published package.
- The validated manual provisioning flow is documented in `docs/VISION_MODEL_PROVISIONING.md`.
- The initial assisted-provisioning foundation for RTMDet detection is already exposed in catalog metadata:
  - upstream checkpoint/config/metafile/paper links
  - planned local builder backend
  - planned supported platforms
  - explicit-consent requirement
- The UI can also trigger installation for models that have an admin-configured source. Supported source env vars are:
  - `TOPOSYNC_VISION_MODEL_SOURCE_<MODEL_ID>`
  - `TOPOSYNC_VISION_MODEL_URL_<MODEL_ID>`
  - `TOPOSYNC_VISION_MODEL_PATH_<MODEL_ID>`
  - `TOPOSYNC_VISION_OFFICIAL_MODEL_SOURCE_DIR`
  - `TOPOSYNC_VISION_OFFICIAL_MODEL_BASE_URL`
- RTMDet detection now also has an experimental assisted local build path:
  - Linux only in this phase
  - requires a local container runtime (`docker` or `podman`) on the processing server
  - still downloads the upstream checkpoint directly on that machine
  - still validates the exported `end2end.onnx` against the manifest checksum
  - now records provenance per job, including actor, accepted upstream sources, builder metadata, and final ONNX sha256
  - still keeps manual upload as the stable fallback path
  - can be started from the model recovery card or the Processing Servers screen
- RF-DETR detection now also has an experimental assisted local build path:
  - supports `linux`, `darwin`, and `windows` hosts in this phase
  - uses a host Python builder (`rfdetr[onnx]`) instead of MMDeploy
  - downloads the upstream checkpoint directly on that machine before exporting the ONNX locally
  - keeps manual ONNX upload available as the stable fallback path
  - is prioritized in the operator UI when nothing is installed and the local builder is actually available on that machine
- Remote download sources are only enabled when the manifest explicitly allows redistribution. The current built-in RTMDet/RTMDet-Ins manifests do not, so the safe first-party flow is local admin-managed copy.
- Product policy: RTMDet and RTMDet-Ins stay on `guided_upload` for now. RF-DETR is available only through assisted local build in this phase; it is not mirrored, bundled, or redistributed by Toposync.
- Product policy: Ultralytics/YOLO is not part of the official first-party vision runtime path.
- The extension now also ships a built-in RTMDet-Ins segmentation shortlist:
  - `rtmdet_ins_tiny`
  - `rtmdet_ins_small`
  - `rtmdet_ins_medium`
- RTMDet manifests use the dedicated `mmdet_rtmdet` parser, letterbox preprocessing, and COCO-80 labels.
- RF-DETR manifests use the dedicated `rfdetr_detr` parser and the official DETR-style `dets` + `labels` ONNX outputs.
- RTMDet-Ins manifests use the dedicated `mmdet_rtmdet_ins` parser and produce real binary mask artifacts.
- The processing server status now exposes:
  - heuristic hardware recommendations
  - runtime upgrade suggestions for CUDA / DirectML bundles
  - a per-task model catalog with availability (`available`, `manifest_only`, `incompatible`)
  - installation capability and progress for models that can be fetched/copied automatically
  - badges such as `recommended`, `fastest`, `best_quality`, `edge`
- `vision.detect` can emit finite per-detection events (`emit_mode="events"`), filter the stream to packets that contain detections (`emit_mode="filter"`), or keep every frame annotated (`emit_mode="annotate"`).
- `vision.track` is now first-party and detector-agnostic: it consumes `payload["vision"]["detections"]`.
- Every `TrackedObject` now carries `camera_id`, and can optionally carry `world_anchor` plus `appearance_embedding_artifact_name` for future multi-camera association work.
- `vision.segment_instances` writes `payload["vision"]["segmentations"]`, attaches mask artifacts when enabled, and exposes the top mask as the semantic image key `mask`.
- `vision.pose_estimate` consumes existing person detections, runs the selected pose model, and publishes named, unclipped landmarks. The `mediapipe_pose_33` manifest provisions a pinned ONNX artifact on the selected processing server.
- `rtmpose_halpe26` is an experimental 2D alternative with an explicit Halpe26 skeleton. Its guided flow requires extracting `end2end.onnx` from the pinned upstream ZIP and uploading the ONNX, not the archive. The upload validates the pinned checksum and backend contract before replacing an existing artifact. Toposync does not bundle, mirror or automatically download these weights.
- RTMPose preserves raw SimCC scores, including values above one; they are not calibrated probabilities. The legacy pose-level score is the source detection score, explicitly identified in metadata, because this model does not output global pose confidence. Halpe26 does not supply relative 3D or a world pointing direction.
- Pose annotations retain their originating detection through `vision.track`; subject-specific processing does not rerun all people. Legacy keypoints remain available.
- `vision.gesture_recognize` adds time-based hand-raised, both-hands-raised, waving and pointing-candidate states. A two-dimensional pointing candidate is not a three-dimensional target or authorization to act.
- Setup, payload semantics, camera geometry requirements and experimental limitations are documented in [Pose, feet and gestures](../../docs/pose-gestos.md). These capabilities are not qualified for unattended physical actions.
- `ModelManifest` now also accepts optional `capabilities` such as `reid`, so future re-identification models can be cataloged without changing the registry shape.
- The pipeline editor now chooses models by task, not framework/vendor id. Basic setup is guided for common users, while advanced details expose runtime/model internals and custom manifest import when needed.
- The first-party tracking backends are:
  - `byte_world` (primary): ByteTrack-style high/low confidence association with optional world-anchor matching.
  - `simple_iou_kalman`
  - `norfair`
- `simple_iou_kalman` and `norfair` remain useful benchmark baselines, but camera presets and recipes should use `byte_world`.
- `vision.track` emits stable per-object lifecycle packets directly. The product identity is `payload["subject"]["id"]`; technical tracker details stay available as `tracklet_id`, `tracklet_ids`, `raw_tracking_id`, and `identity_id`.
- Frame annotations use `payload["vision"]["detections"]`, `payload["vision"]["tracks"]`, and `payload["vision"]["segmentations"]`. Consumers that need one visual subject from an annotated frame should prefer tracks, then detections, then segmentations.
- `vision.group_events` consumes individual `subject.type="event"` packets and emits aggregated `subject.type="group_event"` packets with `group_event_id`, `member_event_ids`, `category_summary`, and grouped `subject.bbox01` for quieter storage and notifications.
- A member CLOSE publishes a group UPDATE with that member inactive immediately, bypassing the periodic update interval. It does not close the group session; consumers can end relations without waiting for group idle timeout. Repeated member CLOSE packets do not publish another update.
- Spatial relations expose `visits` for the currently followed pair: confirmed approaches after observed separation, within `visit_window_seconds` (default 300). Initial proximity is not an observed entry. Membership loss, capture reset, participant change and timeout clear history; short departures within closing grace remain one occurrence. Each group retains at most 256 entries and reports a lower bound on overflow. This does not identify people across tracking sessions or infer intent. New person/vehicle presets include repeated approaches in their notification description; existing custom descriptions remain unchanged.
- Spatial relations require fresh capture evidence from each matched member in the same capture generation. Group publication does not renew member evidence. Dwell advances using the older member media timestamp and pauses when either member is unavailable; this estimates observed continuity and does not certify what happened between frames.
- Group identifiers and streams include the producer lifecycle epoch. A restarted producer creates independent groups even when its display counter starts at 1; it does not inherit proximity duration or assert continuity with the previous producer. Older groups close independently through their lifecycle or the consumer stale timeout. Consumers must treat identifiers as opaque.
- `vision.detect` events are short OPEN/CLOSE notifications; use `vision.track` when you need temporal identity, movement, and long-lived per-object lifecycle.
- `vision.crop_objects` supports `preserve_scene: true` to retain the selected image while attaching the projected subject context; it avoids a second image copy and keeps references outside the subject box visible. Default behavior still crops the subject.
- Object crops derived from `subject.bbox01` carry `subject_context` with subject identity, original and crop-relative bounding boxes, and capture evidence. Crop-relative coordinates use the actual pixel boundaries after rounding, reusing the crop/warp projection path. Detection fallback does not claim subject binding.
- `vision.crop_objects` crops the bbox from `payload["subject"]["bbox01"]` by default and preserves event/subject identifiers. It is not instance segmentation.

## Experimental visual history backend

Automatic installation requests also verify the checksum of an existing artifact before reporting it ready. This runs in the cancellable installation job, without copying or downloading the file again. A mismatch preserves the existing bytes and requires an explicit reinstall; pose models additionally retain their runtime validation.

The AI history endpoint accepts `mode=visual` for multilingual text-to-image ranking of visible notification images. It reuses notification authorization, time filters and SQLite. Similarity does not confirm identity or trigger automation. The notification filter offers words, text meaning and images. The four-file model set is available to the existing model installer as `clip_multilingual_history`; the simplified setup action in the search interface is still pending.

The Vision extension declares the tokenizer and safetensors reader dependencies. Its model manifest groups four pinned artifacts under the installation's `vision-models` directory. Explicit installation uses one cancellable job, verifies every file and reuses intact local artifacts. A failed transfer keeps earlier verified files for a subsequent attempt. Catalog availability requires all four checksums; search never downloads models. The image encoder is [Xenova CLIP ViT-B/32](https://huggingface.co/Xenova/clip-vit-base-patch32/tree/d15189d7028b43f1d3e65039190477f6af591c2a) and the aligned text encoder is [Sentence Transformers multilingual CLIP](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1/tree/58edf8cada9e398793dca955574a48cbb7f18be2). The current quantized text export was validated on ARM64 CPU; other platforms need their own validation.

Search indexes up to eight images per request from the latest 250 matching notifications, with a 64 MiB image scan budget and two inference threads. Sessions are released after each request. The response reports incomplete indexing, unavailable images, scope truncation and fallback to literal text when the optional backend is unavailable. Changed or deleted evidence is rechecked before returning cached results. Model checksums and preprocessing version isolate the vector space. This bounded backend does not certify retrieval accuracy, all historical records, or unattended safety use.

## Future runtime compatibility

- Pipeline configs must stay task-oriented and model-oriented. Do not add runtime, device, delegate, or vendor-specific toggles such as `use_coral` to `vision.detect`, `vision.classify_image`, or related operators.
- New inference stacks should be introduced as optional runtime backends plus `ModelManifest` entries. The manifest owns `runtime`, `artifact_format`, `input.dtype`, hardware accelerators, acquisition metadata, and provenance.
- Do not reuse an existing `model_id` for a materially different artifact/runtime. For example, an Edge TPU/TFLite artifact should get its own model id rather than replacing an ONNX model id.
- Runtime-specific upload, build, and install flows should remain disabled with clear diagnostics until the backend and artifact validation are implemented.
