Metadata-Version: 2.4
Name: matrice_streaming
Version: 0.18.0
Summary: Common server utilities for Matrice.ai services
Author-email: "Matrice.ai" <dipendra@matrice.ai>
License: MIT
Keywords: matrice,streaming,utilities,mypyc,compiled
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: MacOS
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE.txt
Provides-Extra: runtime
Requires-Dist: psutil>=7.2.2; extra == "runtime"
Provides-Extra: dev
Requires-Dist: pytest==9.0.3; extra == "dev"
Requires-Dist: pytest-cov==7.1.0; extra == "dev"
Requires-Dist: pytest-mock>=3.15.1; extra == "dev"
Requires-Dist: pytest-asyncio>=1.4.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.4.0; extra == "dev"
Requires-Dist: fakeredis>=2.37.0; extra == "dev"
Requires-Dist: respx>=0.20; extra == "dev"
Requires-Dist: numpy>=2.5.1; extra == "dev"
Requires-Dist: opencv-python-headless>=4.13.0.92; extra == "dev"
Requires-Dist: pyzmq==27.2.0; extra == "dev"
Requires-Dist: msgpack==1.2.2; extra == "dev"
Requires-Dist: redis>=8.1.0; extra == "dev"
Requires-Dist: psutil>=7.2.2; extra == "dev"
Requires-Dist: pydantic>=2.13.5; extra == "dev"
Requires-Dist: ruff==0.16.0; extra == "dev"
Requires-Dist: bandit[toml]==1.9.4; extra == "dev"
Requires-Dist: mypy==2.3.0; extra == "dev"
Requires-Dist: pre-commit==4.6.1; extra == "dev"
Requires-Dist: pip-audit==2.10.1; extra == "dev"
Requires-Dist: coverage[toml]>=7.16.0; extra == "dev"
Requires-Dist: matrice-common; extra == "dev"
Dynamic: license-file
Dynamic: requires-python

# matrice_streaming — Streaming Gateway KT Docs

This folder is the knowledge-transfer reference for `matrice_streaming`, the
**Streaming Gateway (SG)**. It is written to be walked through in a live KT session:
a presenter takes each chapter top to bottom with a new engineer. Every chapter is
self-contained. Chapters 01–06 follow the same shape — an "In one minute" summary, a
code-free high-level walkthrough, a "Going deeper" section with file/line anchors into
the source, and a configuration table; 07 is a flat reference and 08 an appendix.

## What the Streaming Gateway is

SG is the frame producer of the Matrice video pipeline. Given a set of cameras, it
opens each stream (RTSP fronted by MediaMTX, or file/HTTP sources), decodes frames at
a target FPS, and publishes every decoded frame into a per-camera shared-memory ring
buffer that downstream processes — primarily the inference engine — attach to and read
**zero-copy**. It runs hundreds of cameras per host and manages the camera set fully
dynamically: cameras are added, updated, and removed at runtime without restarting the
process, driven by Kafka events plus a periodic backend poll.

There are two decode backends, chosen at construction:

- **NVDEC (GPU)** — the production path. Demuxed compressed packets are decoded by
  NVIDIA's hardware decoder; frames stay on the GPU as NV12 surfaces and are handed to
  the inference engine through CUDA IPC, never touching host RAM. This path gets the
  depth in these docs (chapter 03).
- **OpenCV (CPU)** — a `cv2.VideoCapture`-based fallback for hosts without a usable
  GPU decode stack. It is covered briefly in chapter 05.

SG does **not** run inference and does not force preprocessing: frames are published
at native resolution and the inference engine owns resize/letterbox/color conversion.

## Where SG sits

```
 Cameras (RTSP via MediaMTX / file / HTTP)
              │
              ▼
 ┌─────────────────────────────────────────────────┐
 │  Streaming Gateway (this package)               │
 │  demux → decode (NVDEC on GPU in prod) →        │
 │  publish into per-camera ring buffers           │
 └─────────────────────────────────────────────────┘
              │  /dev/shm DataBus ring buffers
              │  (NVDEC: CUDA IPC handles to GPU-resident NV12 frames)
              ▼
 Inference Engine ──► results ──► analytics / results-agg
              ▲
              │  control plane
 ┌─────────────────────────────────────────────────┐
 │  Matrice backend: REST API (camera list, Kafka  │
 │  connection info, status) + Kafka (camera       │
 │  add/remove/update events, stop commands,       │
 │  heartbeats, metrics)                           │
 └─────────────────────────────────────────────────┘
```

Three external dependencies, and only three: the backend REST API (via the
`matrice_common` SDK's `Session.rpc`), Kafka, and shared memory via
`matrice_common.stream` (DataBus / CUDA IPC ring buffers). The `matrice_common` SDK
has its own KT doc set in the `py_common` repo; chapter 04 here explains the publish
boundary from SG's side so you do not need that set to follow these docs.

## Session reading order

Read the chapters in order; each builds on the picture established by the previous
one, but none requires flipping back.

| # | Chapter | One line |
|---|---------|----------|
| 01 | [01-architecture-overview.md](01-architecture-overview.md) | The big map: `StreamingAction` → `StreamingGateway` → one decode backend, the process/thread model, lifecycle states, and where every output goes. |
| 02 | [02-camera-management.md](02-camera-management.md) | How the camera set stays in sync at runtime: `InstanceEventListener` (Kafka + periodic poll), the dynamic camera manager, config diffing, and phantom-camera self-healing. |
| 03 | [03-nvdec-pipeline.md](03-nvdec-pipeline.md) | The production decode path in depth: demuxers, codec handling, one worker process per GPU, the decoder pool, GPU placement, and the watchdog. |
| 04 | [04-publishing-and-matrice-common.md](04-publishing-and-matrice-common.md) | The publish boundary: DataBus addresses, ring-buffer semantics, CUDA IPC handle sharing, frame counters, and what SG cleans up (or deliberately leaves) on stop. |
| 05 | [05-opencv-path.md](05-opencv-path.md) | The CPU fallback, briefly: an async worker-process pool sharding cameras, JPEG/BGR publishing through the same DataBus addresses. |
| 06 | [06-metrics-and-heartbeat.md](06-metrics-and-heartbeat.md) | Health reporting: per-camera metrics collection/aggregation, heartbeat payloads, and the Kafka topics the backend watches. |
| 07 | [07-configuration-and-ops.md](07-configuration-and-ops.md) | Environment variables and constructor parameters in one place, plus production symptoms → causes → knobs. |
| 08 | [08-appendix-other.md](08-appendix-other.md) | Everything that is real but not on the main path: `LocalDecoder`, the frame-optimizer seam, camera tampering (Case 1), and other supporting utilities. |
| — | [11-motion-optimizer-test-results.md](11-motion-optimizer-test-results.md) | Motion frame optimizer validation (unit, E2E, threshold tuning). |
| — | [14-camera-tampering-test-results.md](14-camera-tampering-test-results.md) | Blank-screen tampering detector validation on labelled clips + NVDEC E2E. |

Suggested emphasis for a session: 01 and 03 carry most of the weight; 02 and 04 are
where the subtle bugs have historically lived; 05 is a skim; 06–08 are reference.

## Public API in one glance

Everything external lives under `matrice_streaming.streaming_gateway`:
`StreamingAction` (the production entry point — turns an `action_id` into a running,
monitored, auto-restarting gateway), `StreamingGateway` (the orchestrator underneath,
used directly in tests or custom control loops), `InstanceStreamingGatewayUtil` /
`StreamingGatewayUtil` (backend API clients), `InputStream` (the per-camera config
dataclass), and `InstanceEventListener` (Kafka-driven camera reconciliation).
Chapter 01 walks the relationships between them.

## Glossary

- **Stream key** — the gateway's internal handle for one camera stream; maps 1:1 to a
  `camera_id` in the gateway's camera↔stream-key map.
- **Camera config / `InputStream`** — the per-camera configuration dataclass (source
  URL, target FPS, codec, camera ids/keys, location, topic). `width=0`/`height=0`
  means native resolution — SG does not resize by default.
- **Demuxer** — the component that splits a container or transport (RTSP, MP4) into
  raw compressed video packets (H.264/H.265 NAL units) ready for the decoder.
- **NVDEC** — NVIDIA's fixed-function hardware video decoder; the production decode
  backend. Decoding costs essentially no CPU and no CUDA compute.
- **NV12** — a YUV 4:2:0 pixel format (full-resolution Y plane plus interleaved UV at
  half resolution); the native output of NVDEC. As a buffer it is `(H*1.5, W)` uint8.
- **DataBus address** — the name a producer publishes under and a consumer attaches
  to, `{camera_id}__sg__frames`, backed by a `/dev/shm/databus__…` file. Both decode
  backends publish frames under the same address scheme.
- **CUDA IPC** — the CUDA mechanism for sharing a GPU memory buffer between processes
  without copying it to host RAM. On the NVDEC path the shared-memory ring buffer
  carries CUDA IPC handles; the actual NV12 pixels never leave the GPU.
- **Phantom camera** — a camera the gateway believes it is streaming but which has no
  live frame ring buffer in `/dev/shm`. The event listener detects phantoms and
  self-heals by re-adding the camera (chapter 02).
