Metadata-Version: 2.5
Name: cyberwave-edge-core
Version: 0.1.39
Summary: The core component of the Cyberwave Edge Node
Project-URL: Homepage, https://cyberwave.com
Project-URL: Documentation, https://docs.cyberwave.com
Project-URL: Repository, https://github.com/cyberwave-os/cyberwave-edge-core
Project-URL: Issues, https://github.com/cyberwave-os/cyberwave-edge-core/issues
Author-email: Cyberwave <info@cyberwave.com>
License: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: <4.0,>=3.10
Requires-Dist: click>=8.1.0
Requires-Dist: cyberwave>=0.7.5
Requires-Dist: docker>=6.0
Requires-Dist: httpx>=0.25.0
Requires-Dist: rich>=13.0.0
Provides-Extra: build
Requires-Dist: pyinstaller>=6.0.0; extra == 'build'
Provides-Extra: dev
Requires-Dist: hypothesis>=6.100.0; extra == 'dev'
Requires-Dist: mypy>=1.5.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: fuzzy-match
Requires-Dist: cyberwave[fuzzy-match]; extra == 'fuzzy-match'
Provides-Extra: microphone
Requires-Dist: cyberwave[microphone]; extra == 'microphone'
Provides-Extra: ml
Requires-Dist: cyberwave[ml]; extra == 'ml'
Provides-Extra: ml-all
Requires-Dist: cyberwave[ml-all]; extra == 'ml-all'
Provides-Extra: ml-all-audio
Requires-Dist: cyberwave[ml-all-audio]; extra == 'ml-all-audio'
Provides-Extra: ml-all-vision
Requires-Dist: cyberwave[ml-all-vision]; extra == 'ml-all-vision'
Provides-Extra: ml-audio
Requires-Dist: cyberwave[ml-audio]; extra == 'ml-audio'
Provides-Extra: ml-onnx
Requires-Dist: cyberwave[ml-onnx]; extra == 'ml-onnx'
Provides-Extra: ml-stt
Requires-Dist: cyberwave[ml-stt]; extra == 'ml-stt'
Provides-Extra: ml-stt-faster
Requires-Dist: cyberwave[ml-stt-faster]; extra == 'ml-stt-faster'
Provides-Extra: ml-tflite
Requires-Dist: cyberwave[ml-tflite]; extra == 'ml-tflite'
Provides-Extra: ml-wakeword
Requires-Dist: cyberwave[ml-wakeword]; extra == 'ml-wakeword'
Description-Content-Type: text/markdown

<p align="center">
  <a href="https://cyberwave.com">
    <img src="https://cyberwave.com/cyberwave-logo-black.svg" alt="Cyberwave logo" width="240" />
  </a>
</p>

# Cyberwave Edge Core

This module is part of **Cyberwave: Making the physical world programmable**.

Cyberwave Edge Core acts as the orchestrator of Cyberwave edge drivers.

[![License](https://img.shields.io/badge/License-MIT-orange.svg)](https://github.com/cyberwave-os/cyberwave-edge-core/blob/main/LICENSE)
[![Documentation](https://img.shields.io/badge/Documentation-docs.cyberwave.com-orange)](https://docs.cyberwave.com)
[![Discord](https://badgen.net/badge/icon/discord?icon=discord&label&color=orange)](https://discord.gg/dfGhNrawyF)
[![PyPI version](https://img.shields.io/pypi/v/cyberwave-edge-core.svg)](https://pypi.org/project/cyberwave-edge-core/)
[![PyPI Python versions](https://img.shields.io/pypi/pyversions/cyberwave-edge-core.svg)](https://pypi.org/project/cyberwave-edge-core/)
[![Release to PyPI](https://github.com/cyberwave-os/cyberwave-edge-core/actions/workflows/release-pypi.yml/badge.svg)](https://github.com/cyberwave-os/cyberwave-edge-core/actions/workflows/release-pypi.yml)

## Quickstart

SSH to the edge device where you want to install Edge Core, then install the Cyberwave CLI and run the installer:

```bash
# Install the Cyberwave CLI (one-time setup)
curl -fsSL https://cyberwave.com/install.sh | bash

# Run the edge installer (interactive)
sudo cyberwave edge install
```

The installer will prompt you to log in with your Cyberwave account, select a workspace and environment, and persist configuration under `~/.cyberwave/`. You can override the config directory via the `CYBERWAVE_EDGE_CONFIG_DIR` environment variable. Legacy installs that used `/etc/cyberwave` are automatically migrated.

Permissions on the config directory:

- `credentials.json` is written with mode `0600` (owner-only) because it holds your API token.
- `fingerprint.json` is written with mode `0644` (world-readable) because the device fingerprint is a hardware identifier, not a secret. This lets user shells read it even when Edge Core runs as root via `systemd`.
- On `systemd` deployments where Edge Core runs as root, the service re-chowns files under `~/.cyberwave/` on startup so they stay owned by the user whose home directory holds them.

> Don't have a Cyberwave account? Get one at [cyberwave.com](https://cyberwave.com)

### Config files created

The installer and Edge Core create these files in the config directory:

| File               | Description                                 |
| ------------------ | ------------------------------------------- |
| `credentials.json` | API token and workspace information         |
| `fingerprint.json` | Device fingerprint (generated by Edge Core) |
| `environment.json` | Selected environment and twin UUIDs         |
| `edge.json`        | Cached edge device record from the backend (written on registration, refreshed periodically) |
| `{twin-uuid}.json` | Cached twin + asset record (one per twin)   |
| `cameras.json`     | **Deprecated.** Legacy camera device config; superseded by `edge.json` `metadata.cameras` |

Edge Core requires `credentials.json` to operate. `fingerprint.json` is produced by Edge Core; `environment.json` is written by the CLI during setup. `edge.json` is written on registration and stores the full edge device record, including `metadata.cameras` (the device list and the twin-to-device mapping). Every ~5 minutes `reconcile_edge_json_file_sync` fetches the record again, but it rewrites the file only when `metadata.cameras` changed, because a rewrite is what the camera drift check reacts to (see *Camera pins: drift and missing devices*).

## How Edge Core works

On startup (service or direct run), Edge Core performs the following steps:

1. Validate credentials from `credentials.json`.
2. Connect to the backend MQTT broker and verify connectivity.
3. Start a **bootstrap health publisher** that sends periodic edge health messages while drivers are starting up.
4. Register the edge device and record a unique `edge_fingerprint`.
5. Download the selected environment and resolve twins linked to the fingerprint.
6. Start drivers for linked twins. Special handling for attached camera child twins:
   - If a twin is a camera child (has `attach_to_twin_uuid`), Edge Core does not start a separate driver for it.
   - Camera child UUIDs are passed to the parent driver via `CYBERWAVE_CHILD_TWIN_UUIDS`.
7. **Stop the bootstrap health publisher** once drivers are running (drivers publish their own health messages; keeping both would produce duplicate signals in the UI).
8. Pull workflow workers (`wf_*.py`) for the twins listed in `environment.json` and start the worker container (if any worker files exist in `{config_dir}/workers/`).

### Scope of workflow worker sync

Workflow worker sync is scoped strictly to the twin UUIDs the operator selected at install time and persisted to `environment.json` under `twin_uuids`. This is intentionally narrower than the fingerprint-based discovery used for drivers and the bootstrap health publisher: an environment can carry stale `metadata.edge_fingerprint` entries from previous installs, and we don't want those to pull unrelated `wf_*.py` files onto this edge. For backward compatibility, installs that predate the `twin_uuids` field still fall back to fingerprint-based discovery.

During driver startup, Docker image pull progress is mirrored into the edge-core service logs and forwarded through the same MQTT-backed driver log stream used for runtime container logs, so users can follow image download progress remotely.

### Remote restart (Edge REST API)

Request a remote restart of Edge Core via the REST API:

```http
POST /api/v1/edges/{uuid}/restart-core
```

The API will publish an MQTT message to the edge's command topic:

Topic: `edges/{edge_uuid}/command`

Example payload:

```json
{ "command": "restart_edge_core" }
```

When Edge Core receives this command it performs a graceful restart consisting of:

1. Stopping the worker container (if running).
2. Removing cached twin JSON files from the edge config directory.
3. Stopping and removing any edge-managed driver containers, then pruning stopped containers.
4. Resolving any active `driver_starting` alerts on the affected twins so leftovers from the previous run do not stay visible after the restart.
5. Re-downloading the selected environment and restarting drivers.
6. Restarting the worker container (if worker files exist).

The restart deletes the cached twin JSON files without a final push, so an edit a driver made to its twin file after the last runtime tick (see *Twin JSON file*) is lost; everything else is downloaded again from the backend.

Each driver startup attempt creates a `driver_starting` twin alert that tracks the in-flight startup (image pull, container launch, post-launch health probe). The alert is automatically resolved once the driver container is observed `running`; if the attempt fails, it's annotated with the failure phase (e.g. `pull_failed` / `pull_timeout` / `pull_oserror`), escalated to `critical` severity, and resolved. The same lifecycle applies to the ML worker via the `worker_starting` alert, which is raised *before* the image pull and held until the worker container is observed `running` — so a worker whose image is already on disk still reports that it is starting, rather than coming up silently. The alert is therefore guaranteed to clear when the driver has restarted; longer-lived failure conditions are surfaced through separate `driver_start_failure` alerts created by the orchestrator.

While `docker pull` is running, edge-core writes pull progress directly onto the alert's `metadata` (`progress_percent`, `downloaded_bytes` / `total_bytes`, `downloaded_human` / `total_human`, `phase`) so the workbench renders e.g. `Downloading driver image (cyberwaveos/ugv-driver:dev) — 43%` instead of a stale "Still pulling" spinner. The byte counts join the message once every layer has declared its size (`745 MB of 1.55 GB`); until then the image size is not yet known and only the percentage is shown. The same summary feeds the systemd watchdog (`systemctl status cyberwave-edge-core`) and extends `TimeoutStartSec` while bytes are still arriving. The parser, fan-out worker, and tests live in [`cyberwave_edge_core/driver_logs.py`](cyberwave_edge_core/driver_logs.py) and [`tests/test_docker_pull_progress.py`](tests/test_docker_pull_progress.py).

Startup is reported as three distinct stages. `overall_progress_percent` — the number a progress bar should read — spans all three as a single 0–100 that only moves forward (download 0–45, install 45–80, container 80–100), because the card renders one bar and a stage restarting its own count reads as the bar rewinding. `stage_progress_percent` carries the same startup as a per-stage 0–100, for a stage-aware bar to use; the alert's text names the stage on screen, so a stage that reads 100 % is one you can check against what you are being told. The bands are sized by how long each stage takes rather than by how many steps it has, so install — the stage that outlasts the download — gets a comparable share of the bar.

**Downloading** is driven by layers fetched rather than by bytes: docker only declares a layer's size when it starts fetching it, so a byte ratio has an unknown denominator for most of the pull and reads 100 % every time the in-flight batch drains. The layer currently downloading counts as its byte fraction, so the number still moves between layers. 100 % is reserved for a pull docker has confirmed finished. **Installing** — verifying and decompressing, which on a multi-GB image outlasts the download — tracks layer completion (`layers_complete` / `layers_total`). **Starting** covers container create → run → probe.

#### Local driver builds are never pulled

A driver image whose tag is `local`, starts with `local-` or ends with `-local` is treated as a developer build. When such an image already exists on the host, Edge Core uses it as it is: `_use_local_build` takes it out of the parallel pull, the startup alert records `phase: "pull_skipped"`, and the log says `Using the local build of <image>; local-only tags are not pulled`. Before this rule a restart pulled the tag like any other, and because Docker Hub can carry an image with the same tag, the pull quietly replaced the build you were testing. A local-only tag that is not on the host is still pulled, and digests (`@sha256:…`) never count as local. The worker has its own, broader rule; see *Worker image refresh policy*.

#### Which image a twin's driver runs

The `docker_image` in the twin's `metadata.drivers` is what runs, whatever tag it carries (`humble-dev-sha-<sha>`, `humble-0.1.3-rc.7`, `humble-pr-4549-sha-<sha>`, a `@sha256:` digest, a release such as `jetson-humble-0.1.23`). After `docker ps` on the edge, the driver's own services show that reference, not a moving channel tag.

A driver image can declare a stack of sibling images in its `io.cyberwave.driver.compose.*` label (the Go2 declares `go2-ros2-driver`, `ros2-slam` and `ros2-nav2`). Edge Core resolves that graph as follows:

1. **Own services.** A service of the same repository as the twin's image (`plant` and `bridges` for the Go2) runs the twin's reference exactly. The label names it `<repo>:${CW_CHANNEL_TAG}`, which used to resolve to the channel the image was built in, so a twin on `humble-dev-sha-<sha>` ran `humble-dev`. See `_run_the_twin_image_in_its_own_services` in `cyberwave_edge_core/startup.py`.
2. **Siblings of a pinned graph.** CI pins every sibling to an immutable `-sha-` tag and marks it `x-cyberwave.pinned_from` (`pin-driver-compose-images`, `.github/scripts/pin_driver_compose.py`). Edge Core runs those tags as written. That is what keeps driver and siblings on one release.
3. **Siblings of an unpinned graph.** If the twin's image is a release tag (`<distro>-X.Y.Z`, optionally `jetson-` led), the `${CW_CHANNEL_TAG}` siblings are requested at that same tag; otherwise they follow the image's `io.cyberwave.image.channel-tag` label. See `_resolve_channel_tag` in `cyberwave_edge_core/driver_selection.py`. PR re-tags (`humble-pr-…`), dev prereleases (`humble-0.1.3-dev.5`) and moving tags are not release tags and keep following the label. Siblings keep their own version counters, so `ros2-slam:jetson-humble-0.1.23` is generally not a published image: step 4 refuses this case rather than letting those services fail to pull. The simulator reads the same graph through `get_image_declared_services` but has no step 4, so a release-tagged twin there fails to pull those siblings instead of being refused.
4. **No proof of one release, no start.** A sibling that is still declared by placeholder after step 2 is whatever its channel resolves to, so the stack may mix releases, or name an image that does not exist. Moving tags (`humble-dev`) and version-shaped tags that came from step 3 both count: the check looks at how the graph named the sibling, not at the shape of the tag. Edge Core refuses to start that twin: the log has `Twin '<name>' refused: driver image … does not pin its sibling services to a release … : slam=<image>`, and a `driver_start_failure` alert titled **Driver stack is not pinned to one release** names the services. Only drivers with sibling images are affected, and only a twin that names one build (a release, a `-sha-` tag, a digest): a twin on a moving tag such as `humble` asks for a channel, its siblings follow the same channel, and it starts as before.

Ways to run such a twin: use a driver release whose images are pinned; pin each named service in the twin's `service_image_overrides` (a tag or a `@sha256:` digest; a service pinned there counts as pinned); or set `CYBERWAVE_DRIVER_CHANNEL_TAG` on the edge to pin the whole graph, driver services included, to one train (an operator override; it disables steps 1 and 4).

`service_image_overrides` is applied last and always wins. An image without a registry namespace (a bare local build name) is never refused, because there is nothing to tell its siblings from third-party images such as redis.

Tests: `tests/test_image_declared_topology.py` and `tests/test_resolve_twin_driver_specs.py`.

#### When `docker create` times out

`docker create` runs with a timeout (`CYBERWAVE_DRIVER_CREATE_TIMEOUT_SECONDS`, 30 s by default), but the timeout only kills the CLI. A busy Docker daemon usually goes on to finish the create, which used to leave a container in `Created` that Edge Core had never registered and therefore never retried. Edge Core now waits up to 10 more seconds for the container to appear (`_wait_for_container` in `docker_launch.py`). If it does, the launch carries on as normal, with a warning in the log; only if it does not is the attempt failed with `docker_create_timeout`.

## Model Manager (ML model cache)

Edge Core includes a `ModelManager` that resolves ML model weights into a local cache before starting the worker container. It is designed for both online deployments and air-gapped sites: it prefers fresh weights from Cyberwave when the network is available, falls back to upstream public mirrors, and finally to whatever is already on disk.

**Cache location:**

| Platform | Default path |
|---|---|
| All | `~/.cyberwave/models/` |

Override with `CYBERWAVE_EDGE_CONFIG_DIR`.

**Cache layout:**

```
<cache_dir>/
├── manifest.json            # index of all cached models
├── yolov8n/
│   ├── yolov8n.pt           # weight file
│   └── metadata.json        # checksum, runtime, source URL, upstream URL
└── background-subtraction/
    └── ...
```

**Model requirements discovery:** Edge Core scans `*.py` files in `~/.cyberwave/workers/` for `cw.models.load(...)` calls to determine which weights to ensure.

`~/.cyberwave/models/` and `~/.cyberwave/workers/` are created eagerly on Edge Core startup (even before any worker runs), with ownership matching the invoking user, so operators can drop pre-staged weights into `~/.cyberwave/models/{model_id}/` from a regular shell.

On `systemd` deployments where Edge Core itself runs as root, every per-model staging directory, downloaded weight, sidecar `metadata.json`, and the top-level `manifest.json` is also chowned to the invoking host user at write time — not just at startup. This keeps the cache both readable and **writable** from inside the worker container (which launches with `--user $UID:$GID`); without it, `cw.models.load` crashes with `PermissionError: [Errno 13]` on `iterdir` when traversing a root-owned `cache_dir/{model_id}/`. Writability matters because the SDK's `UltralyticsRuntime` sandboxes lazy secondary-asset downloads (e.g. the MobileCLIP text encoder for YOLOE / YOLO-World text prompts) into the weight directory. The wiring lives in `ModelManager(owner_uid_gid=resolve_config_owner_uid_gid())`.

### Resolution order

For each required model, `ensure_model(model_id)` runs the following steps:

1. **Reconcile disk.** If `cache_dir/{model_id}/` already contains a weight file (with or without a sidecar), it is registered in the manifest. A missing sidecar is generated from the on-disk file (with a freshly computed SHA-256) and tagged `downloaded_from: prestaged`. This is how an operator pre-stages weights from a USB stick on an air-gapped site.
2. **Verify cache integrity.** If the local file's SHA-256 matches the manifest checksum, the cache is *intact*.
3. **Pre-staged short-circuit.** When an intact entry is tagged `downloaded_from: prestaged`, Edge Core returns it without ever contacting the catalog. Pre-staged files are operator-curated truth; to force a re-download, evict the model directory.
4. **Best-effort catalog probe.** For non-prestaged intact entries, Edge Core does a short-timeout `GET /api/v1/mlmodels/...` to compare checksums.
   * Catalog unreachable, no checksum, or matching checksum → return the cached file (no download).
   * Catalog returns a different checksum → fall through to the download path.
5. **Download.** Sources are tried in priority order:
   1. **Cyberwave-hosted signed URL** from `GET /api/v1/mlmodels/{uuid}/weights` — used for checkpoints we have uploaded to our private GCS bucket (e.g. internally trained or mirrored models). Authenticated, served from infrastructure we control.
   2. **Upstream weights URL** from the catalog entry (`download_url` / `metadata.upstream_weights_url`) — used for community checkpoints we did not mirror.
   3. **Runtime-managed download** as a final fallback: when neither URL is set but the catalog entry's `edge_runtime` ships its own weight resolver (today: `ultralytics` against its GitHub releases hub), Edge Core invokes it in a subprocess and caches the resulting file as `downloaded_from: runtime_managed`. This is what lets `.pt` catalog entries (YOLO26, YOLOE-26) work without a mirrored upstream URL. Requires the runtime to be installed on the edge host — see [ML runtime extras](#ml-runtime-extras).

   The first source that yields a checksum-verified file wins. The sidecar records `downloaded_from` (`artifact_url` / `download_url` / `runtime_managed` / `prestaged`), `source_url` (the public URL we fetched, or `null` for artifact and runtime-managed downloads — neither is a re-fetchable URL we want to persist), and `upstream_url` (provenance).
6. **Fail-soft.** If every download attempt fails *and* the cached file is intact, Edge Core returns the cached path with a warning. This keeps workers running across transient network failures and on permanently air-gapped sites. If the cache is empty or corrupt, `RuntimeError` is raised.

**Cache integrity:** SHA-256 checksums are verified on every `ensure_model` call when a checksum is recorded — on cold start, after every download, during disk reconciliation, and on every warm-cache hit. There is no shortcut. For multi-gigabyte checkpoints this is the dominant cost of the call, but `ensure_model` only runs when a worker file changes (rare) or the worker container restarts; in exchange we get unconditional bit-rot detection. A checksum mismatch on a downloaded artifact triggers a re-download attempt; a download whose checksum does not match the catalog is rejected and the partial file removed.

### Pre-staging weights for air-gapped deployments

On a site without internet access, an operator can place weights directly into the cache:

```bash
mkdir -p ~/.cyberwave/models/yolov8n
cp /usb-stick/yolov8n.pt ~/.cyberwave/models/yolov8n/
```

Edge Core picks up the file on the next `ensure_model("yolov8n")` call, computes a SHA-256, and writes a sidecar `metadata.json` so subsequent runs are deterministic. The `runtime` field is inferred from the file extension (`.pt` → `ultralytics`, `.onnx` → `onnxruntime`, `.engine`/`.trt` → `tensorrt`, `.tflite` → `tflite`, `.pth` → `torch`, `.xml` → `opencv`); provide a hand-written `metadata.json` to override.

**Updating in place.** Operators can drop a new build into the same directory and Edge Core will detect the change on the next call:

```bash
cp /usb-stick/yolov8n-v2.pt ~/.cyberwave/models/yolov8n/yolov8n.pt
```

The mismatch between the on-disk SHA-256 and the manifest checksum triggers a re-stamp (not a re-download), provided the sidecar still records `downloaded_from: prestaged`. This keeps offline edges functional across model upgrades. Files that were previously *downloaded* by Edge Core keep the corruption-detection semantics — bit-rot still triggers a re-download attempt rather than being silently accepted.

**Pre-staged files are never auto-overwritten by catalog updates.** Once a file lives under `cache_dir/{model_id}/` with a `downloaded_from: prestaged` sidecar, Edge Core treats it as the source of truth and skips the catalog probe entirely. To force a re-download from the Cyberwave catalog, evict the model:

```bash
rm -rf ~/.cyberwave/models/yolov8n
```

Provide a hand-written `metadata.json` (with `filename`, `checksum_sha256`, and `runtime`) when there are multiple weight files in the directory or when corruption detection should compare against a known-good hash.

### Self-healing the model cache

A failed download leaves an empty `cache_dir/{model_id}/` directory behind — the `mkdir` on the download path runs *before* the network fetch, and any error in the downloader (HTTP timeout, missing upstream weight, runtime self-download crash) skips the cleanup. Without intervention, the SDK in the worker container would later route that directory into `torch.load` and crash with `IsADirectoryError` on every restart.

Edge Core handles this in three layers:

* **Per-download cleanup** (conservative). If `_download_model` or `_download_runtime_managed` raises, the staging directory is removed before the exception propagates, provided it contains only an `.dl_*.part` partial-download file, a `metadata.json` sidecar, or nothing at all. Operator-staged content blocks the prune.
* **Startup sweep** (conservative). `ModelManager()` walks `cache_dir/` once at construction time and prunes any per-model subdirectory that is not tracked in the manifest *and* contains only orphan cruft. This catches hosts that were already wedged before this code shipped. Operator-staged content blocks the prune.
* **`evict_model('foo')`** (decisive). When there is no manifest entry but `cache_dir/foo/` exists on disk, the directory is removed **unconditionally** and `True` is returned. This is the recovery hook for a remote eviction API to call without shell access to the host; an explicit `evict` is treated as a stronger operator-intent signal than the conservative auto-sweeps.

For the two conservative layers, a directory containing an unexpected file (an operator's `README.md`, a hand-staged weight file with an unusual name, a sub-directory) is preserved untouched — "human always wins". Use `evict_model` explicitly when you mean to clear the slot regardless of contents.

### ML runtime extras

The default `pip install cyberwave-edge-core` keeps the runtime surface lean — no Torch, no Ultralytics, no ONNX. That's enough for workflows that resolve weights through a backend-hosted URL or pre-staged files. Catalog entries that rely on the **runtime-managed download** fallback (step 5.3 above) need the matching runtime installed on the edge host:

```bash
# Pull in Ultralytics so YOLO/YOLOE .pt entries can self-download
pip install 'cyberwave-edge-core[ml]'

# Other available extras (mirror the cyberwave SDK extras 1:1):
pip install 'cyberwave-edge-core[ml-onnx]'       # onnxruntime
pip install 'cyberwave-edge-core[ml-tflite]'     # tflite-runtime
pip install 'cyberwave-edge-core[ml-wakeword]'   # openwakeword + onnxruntime
pip install 'cyberwave-edge-core[ml-stt]'         # pywhispercpp (Whisper.cpp GGML)
pip install 'cyberwave-edge-core[ml-stt-faster]'  # faster-whisper (CTranslate2)
pip install 'cyberwave-edge-core[ml-audio]'      # silero-vad + torchaudio (Audio Assistant)
pip install 'cyberwave-edge-core[fuzzy-match]'   # rapidfuzz (Fuzzy Matcher)
pip install 'cyberwave-edge-core[microphone]'    # host mic capture (Audio Track trigger)
pip install 'cyberwave-edge-core[ml-all-vision]'  # YOLO / ONNX / TFLite only
pip install 'cyberwave-edge-core[ml-all-audio]'   # all speech extras (avoid on Pi — pick ml-stt / ml-stt-faster per model)
pip install 'cyberwave-edge-core[ml-all]'         # CI/dev fat bundle only
```

When a required runtime is missing, the resolver raises a `RuntimeError` that spells out the three workarounds: drop the weight file into `~/.cyberwave/models/{model_id}/`, upload to `/api/v1/mlmodels/{uuid}/weights`, or set `metadata.download_url` on the catalog entry.

On a **frozen PyInstaller binary** (`cyberwave edge install --channel dev|staging`), `sys.executable` is the edge-core CLI wrapper rather than a Python interpreter, so the runtime-managed downloaders cannot spawn the subprocess they need. In that case `ensure_model` raises `_RuntimeManagedDeferred`, which `ensure_models` treats as **non-fatal**: the model is omitted from `last_ensure_failures` and the worker container — which bundles the runtime — resolves the weight on its first inference call. This prevents a misleading `model_download_failure` alert from firing while the workflow runs fine (CYB-2103).

## Worker container

Edge Core manages one ML worker container per edge device (container name: `cyberwave-worker-{env_uuid[:8]}`). The worker container runs Python worker scripts from the local workers directory and has access to cached model weights.

### Worker directory layout

Place worker scripts in `{config_dir}/workers/` (default: `~/.cyberwave/workers/`):

```
~/.cyberwave/
├── workers/
│   ├── detect_people.py        # Custom worker
│   └── cyberwave.yml           # Optional: list model requirements
└── models/                     # Auto-managed model cache
    ├── manifest.json
    └── yolov8n/
        └── yolov8n.pt
```

### cyberwave.yml

Optionally declare model requirements so Edge Core can pre-download them before starting the worker container:

```yaml
models:
  - yolov8n
  - background-subtraction
```

Edge Core also auto-detects models by scanning `cw.models.load("...")` calls in worker Python files.

### Worker container environment variables

| Variable | Value |
|---|---|
| `CYBERWAVE_API_KEY` | Injected from credentials |
| `CYBERWAVE_ENVIRONMENT_UUID` | Active environment UUID |
| `CYBERWAVE_TWIN_UUIDS` | Comma-separated twin UUIDs in environment |
| `CYBERWAVE_DATA_BACKEND` | `zenoh` |
| `ZENOH_CONNECT` | Set when a Zenoh router is configured |
| `ZENOH_SHARED_MEMORY` | `false` by default (opt-in; requires `--ipc=host`) |

### File watching and hot-reload

Edge Core monitors `{config_dir}/workers/` every reconcile cycle (~15 seconds). When `.py` files are added, removed, or modified, Edge Core automatically:

1. Re-scans model requirements.
2. Pre-downloads any missing models.
3. Restarts the worker container with the updated files.

A minimum cool-down of 10 seconds between successive automatic restarts prevents rapid churn when files are written incrementally (e.g. by `rsync` or `scp`).

### Workflow-driven worker lifecycle

The worker container is brought up and torn down based on whether any active workflows are currently synced for the connected twins:

- **Startup:** after pulling worker files from the backend (step 8), Edge Core inspects `{config_dir}/workers/`. If at least one `wf_*.py` file is present, the worker container is started; otherwise it is left down — and the `cyberwaveos/edge-ml-worker` image is **not** pulled.
- **Periodic reconcile:** every ~5 minutes (configurable via `CYBERWAVE_WORKER_SYNC_INTERVAL_LOOPS`), Edge Core resyncs worker files from the backend. If a workflow was activated mid-run and new files appeared, the worker container is started. If every workflow was deactivated and the directory is now empty, the container is stopped. Both calls are idempotent.
- **Immediate reconcile on activate:** when a `run_on_edge` workflow is activated (UI, CLI, or API), the backend publishes a `sync_workflows` command on `cyberwave/twin/{twin_uuid}/command` for each twin the workflow references. Edge Core runs `reconcile_worker_sync` right away so the new `wf_*.py` lands within seconds instead of up to one periodic interval. When the worker container is already running, Edge Core also bounces it via `WorkerManager.restart` so the new modules are actually imported — `start()` is a no-op for a running container, so without this step activation would only take effect after the next file-watch tick (or a manual `cyberwave worker restart`). The bounce is skipped when the container itself started *after* the newest change in `{config_dir}/workers/` — it already imported those files on the way up, so restarting would only discard a boot in progress and delay the workflow going live. Anything that cannot be determined (a failed `docker inspect`, an unreadable workers directory) falls back to bouncing. Failures fall back to the periodic reconcile; the MQTT nudge is best-effort, not a correctness guarantee.
- **Immediate reconcile on remove:** when a `run_on_edge` workflow is deactivated, soft-deleted, or has its `run_on_edge` flag flipped off, the backend publishes a surgical `remove_workflow_worker` command on the same twin command topic, naming the specific worker filename(s) to unlink. Edge Core unlinks the matching `wf_*.py` from `{config_dir}/workers/` and immediately reconciles the worker container lifecycle, so deactivation propagates within seconds rather than waiting up to one periodic interval (~5 minutes) plus the bulk-sync two-strikes cleanup cycle. For partial deactivations (sibling workflows still active), the same in-place restart fires so the removed worker is actually unloaded. The same nudge also covers workflows soft-deleted as part of a workspace or environment teardown. The periodic reconcile remains the correctness backstop if the MQTT publish is dropped.
- **MQTT command dedupe:** both `sync_workflows` and `remove_workflow_worker` carry a `request_id`; Edge Core keeps a bounded FIFO of the last 1024 IDs it has handled per process and drops repeats. QoS retries and broker re-publishes therefore don't spawn duplicate sync threads or repeat unlink operations.
- **Sync errors:** if a sync cycle reports any errors, the lifecycle reconcile is skipped to avoid churning a healthy worker on transient API failures, and stale-file cleanup is skipped too. A twin whose fetch failed reports no workflows, which is indistinguishable from a genuine deactivation, so deleting on that picture would let an outage, an expired token or a DNS blip spanning two sync periods unlink every `wf_*.py` on the device. Deletion needs a complete picture; the next successful sync re-evaluates state.

#### How this is tested

Reconcile branching, watcher debouncing, and MQTT command handling are unit-tested with mocked Docker in `tests/test_worker_lifecycle_reconcile.py`, `tests/test_worker_watcher.py`, `tests/test_worker_health.py`, and `tests/test_startup_remove_workflow_worker.py`. `ResourceLimits` argv assembly is covered in `tests/test_worker_health.py` (`TestResourceLimits`).

**Tier 1 — scripted lifecycle integration** (`tests/integration/test_worker_lifecycle_docker.py`, marker: `docker`): 14 deterministic phases exercise the real Docker daemon — cold start, add/remove worker files, restart-in-place, surgical remove, edge-core process restart, cooldown enforcement, image pull policy, `request_id` dedupe for both `remove_workflow_worker` and `sync_workflows`, health-monitor `record_stop` wiring, spontaneous-exit detection, hot-reload through `WorkerWatcher.force_restart`, and a full activate→deactivate→reactivate operator cycle. The suite auto-skips when Docker is unavailable and uses a session-built `cyberwave-worker-stub` image so it never pulls the real `cyberwaveos/edge-ml-worker`.

**Tier 2 — Hypothesis chaos** (`tests/integration/test_worker_lifecycle_chaos.py`, marker: `docker`): a `RuleBasedStateMachine` drives the same surface (`WorkerManager`, `reconcile_worker_lifecycle`, `_run_remove_workflow_worker`, `WorkerWatcher`) with randomized operation interleavings against the real Docker daemon, asserting eight cross-cutting invariants after every step (state consistency, no spurious `docker rm`, container-name stability, `request_id` dedupe, immutable-tag pull-at-most-once, `record_stop` count matches `running→exited` transitions, cooldown respected, no orphan containers). PR CI runs with `max_examples=20, stateful_step_count=30, derandomize=True` (deterministic, ≤ 3 min); staging also runs a lighter chaos slice (`max_examples=5`, `stateful_step_count=15`). The nightly workflow (`.github/workflows/edge-core-nightly-chaos.yml`) mirrors the PR `test_edge_core` job (Tier 1 Docker integration, Tier 3 + Tier 3-ML shell E2E against a live backend) and adds Tier 2 chaos with `max_examples=200, stateful_step_count=100` and a random seed; it files a GitHub issue on failure without blocking PRs. Monotonic timing for cooldown and health windows goes through `cyberwave_edge_core._clock.now_monotonic()` so tests can advance time without sleeping.

**Tier 3 — Full E2E against a live backend** (`cyberwave-edge-nodes/test_worker_lifecycle_e2e.sh`): closes the gaps left by Tier 1/2 by exercising the complete workflow activation pipeline: creates real backend objects (project → environment → twin → workflow with manual-trigger + send-MQTT nodes) via the HTTP API, activates the workflow, and asserts that the `sync_workflows` MQTT message is delivered to a running edge-core, that edge-core fetches `worker_source` from the `/edge-sync` endpoint, writes the resulting `wf_*.py` file, and starts the worker container. Deactivation is then exercised and the container stop is asserted. Runs after `test_edge_core_and_cli.sh` in the same CI job so the backend and broker are already healthy. The test is self-contained: it builds a minimal `cyberwave-worker-stub:it` (Alpine `sleep infinity`) and a `cyberwave-edge-test` image, and tears both down on exit.

**Tier 3-ML — Model download and inference E2E** (`cyberwave-edge-nodes/test_ml_inference_e2e.sh`): extends the Tier 3 test with two focused phases. Phase A creates a `camera_frame → call_model(yolov8n.onnx)` workflow, activates it, and asserts that edge-core pre-downloads `yolov8n.onnx` (~6-12 MB, YOLOv8 Nano ONNX from the Cyberwave CDN) via `ModelManager.ensure_models()` before starting the worker container. Phase B starts `cyberwaveos/edge-ml-worker:local` and runs `onnxruntime` inference on a synthetic 640×640 image, asserting clean exit and a valid 3-D output tensor. Requires `cyberwaveos/edge-ml-worker:local` to be pre-built (see the script header for the `docker buildx build` command); the CI step skips gracefully when the image is not available.

### Worker image refresh policy

`WorkerManager._ensure_image_pulled` decides whether to issue `docker pull` before each worker (re)start:

| Tag basename (with optional `-gpu`/`-cpu`/`-arch` suffix) | Mutability | Pull behaviour |
|---|---|---|
| `latest`, `dev`, `staging`, `nightly`, `edge`, `main`, `master` | Mutable | Pull every time, even when the image is already present locally. If the registry is unreachable but a local copy exists, fall back to the local copy and warn. |
| Anything else (`v1.2.3`, dated build IDs, `@sha256:…`, `local`) | Immutable | Skip the pull when the image is already present locally; only pull when missing. |

This avoids the previous failure mode where a stale `cyberwaveos/edge-ml-worker:dev-gpu` image stayed cached after a developer pushed a new build — operators no longer need to remember to `docker rmi` before restarting the worker. Immutable tags keep the original fast-path so versioned production deployments are not slowed down by an extra round-trip to the registry on every restart.

`local` counts as immutable because nothing publishes it: a `:local` image is whatever the last local `docker build` produced, so a pull can only fail — a wasted round-trip, or a connect timeout on an offline machine. A `:local` tag with no local copy is still pulled, like any other missing image.

Driver images follow a narrower rule. They are pulled on every start unless the tag is `local`, `local-*` or `*-local` and the image is already on the host (see *Local driver builds are never pulled*). The reason is the same: a local build must survive a restart.

### Pinning a custom worker image (`CYBERWAVE_WORKER_IMAGE`)

`resolve_worker_image()` consults `CYBERWAVE_WORKER_IMAGE` before falling back to the `CYBERWAVE_ENVIRONMENT`-derived tag. This is the worker-side counterpart to the `driver_overrides` field in `credentials.json` for camera/asset drivers, and is the recommended way to run a locally-built or hot-patched worker image without re-pushing to the registry.

Two ways to set it (both honoured by `get_runtime_env_var`):

- **Operator config (preferred for dev hosts, no `sudo`)** — add to the `envs` block of `~/.cyberwave/credentials.json`:

  ```json
  {
    "envs": {
      "CYBERWAVE_ENVIRONMENT": "dev",
      "CYBERWAVE_WORKER_IMAGE": "cyberwaveos/edge-ml-worker:local"
    }
  }
  ```

- **Systemd dropin (preferred for managed deployments)** — `sudo systemctl edit cyberwave-edge-core`:

  ```ini
  [Service]
  Environment=CYBERWAVE_WORKER_IMAGE=cyberwaveos/edge-ml-worker:local
  ```

Either way, restart edge-core (`sudo systemctl restart cyberwave-edge-core`) to reload the resolver.

Use the `:local` tag (no `-gpu`/`-cpu`/`-jetson`/`-hailo` suffix) — same convention as the camera-driver `:local` tag. `_run_container` auto-appends the accelerator suffix (`-jetson` on Jetson hosts, `-gpu` on discrete-NVIDIA hosts, `-hailo` on Pi 5 + AI HAT+) for `cyberwaveos/edge-ml-worker:*` overrides, so the operator only commits and pins one tag.

Typical hot-fix loop (mirroring the camera-driver `:local` tag pattern):

```bash
# 1. Hot-patch the running container (e.g. swap an SDK runtime file).
WORKER=cyberwave-worker-<fingerprint>
docker cp /path/to/patched/file.py "$WORKER:/usr/local/lib/python3.12/dist-packages/cyberwave/.../file.py"
docker exec -u root "$WORKER" rm /usr/local/.../file.cpython-312-x86_64-linux-gnu.so

# 2. Snapshot the patched container as a local-only tag (and an alias on the
#    accelerator-suffixed tag matching the host, so the same image is used
#    regardless of which path `_run_container` resolves to on this host).
docker commit "$WORKER" cyberwaveos/edge-ml-worker:local
docker tag    cyberwaveos/edge-ml-worker:local cyberwaveos/edge-ml-worker:local-gpu
# On Jetson hosts, alias -jetson instead of -gpu:
# docker tag  cyberwaveos/edge-ml-worker:local cyberwaveos/edge-ml-worker:local-jetson

# 3. Tell edge-core to use it (see the two options above), then restart it.
#    Pull will fail (registry has no :local) and `_ensure_image_pulled`
#    falls back to the locally-present image.
sudo systemctl restart cyberwave-edge-core
```

The patched image survives every `docker rm`/`docker run` cycle from edge-core's reconcile loop. To revert, remove the env var (or `systemctl revert cyberwave-edge-core` if you used the dropin) and `docker rmi cyberwaveos/edge-ml-worker:local cyberwaveos/edge-ml-worker:local-gpu` (or `…local-jetson` on Jetson hosts).

When overriding to a registry outside `cyberwaveos/edge-ml-worker:*` (e.g. an internal mirror), include the accelerator suffix (`-gpu` / `-jetson` / `-hailo`) yourself if you need accelerator access — the auto-suffix path in `_run_container` only triggers for the canonical `cyberwaveos/edge-ml-worker:` prefix.

### Worker startup probe

After `docker run --detach`, Edge Core polls the container status for up to 30 seconds (configurable via `CYBERWAVE_WORKER_STARTUP_PROBE_SECONDS`) waiting for it to reach `running` state. The probe uses `docker inspect` to check the full container state, including restart count:

- **`running`**: success — container is up.
- **`exited`/`dead` with no prior restarts**: definitive failure — an alert is sent to all linked twins.
- **`exited` with `RestartCount > 0`**: Docker's restart policy (`--restart=unless-stopped`) is active — the probe keeps waiting.
- **`created`/`restarting`/other transient states**: Docker is still setting up the container — the probe keeps waiting.
- **Probe window elapsed**: the container may still come up (Docker keeps trying via its restart policy), so Edge Core returns success and sends a `worker_start_failure` alert for visibility. The health monitor and periodic reconcile loop track the eventual outcome.

This resilient approach ensures that slow container startups on resource-constrained devices (e.g. Raspberry Pi) do not block the rest of Edge Core's boot sequence.

### Worker readiness

A `running` container is not yet a usable worker. The process inside still has to import your worker modules (downloading model weights the first time), connect to MQTT, subscribe its hooks and run warm-up inference — minutes on a cold device.

The worker reports each of those milestones to `/run/cyberwave/worker-status.json`, which Edge Core mounts into the container and reads once per reconcile cycle (every 15 s). The `worker_starting` alert stays open across the whole sequence, so the dashboard shows `Loading models` or `Warming up models — 2 of 5 models` instead of going quiet the moment the container starts, and a workflow run dispatched to the device explains what it is waiting for rather than appearing stalled.

- **`ready`**: the alert resolves.
- **Container exits during startup**: the alert reports the failure and the phase it reached.
- **`CYBERWAVE_WORKER_READY_TIMEOUT_SECONDS` elapsed** (default 15 minutes): the alert reports that the worker took too long to load its models.
- **No status file within 60 seconds**: the worker image predates status reporting, so Edge Core falls back to treating a running container as started.

The readiness timeout is measured from the container's own start timestamp rather than from when Edge Core began watching, so a long image pull does not eat into the worker's own startup budget and a container restart resets the clock by itself.

The progress card does not survive an Edge Core restart, though. A graceful restart stops the worker container first, so the next start opens a fresh card. But if Edge Core is killed while the worker container keeps running — an out-of-memory kill, a watchdog reset, a crash loop — the card is resolved on the next boot and is not re-opened for that container, and the worker finishes loading with nothing on screen.

`cat /run/cyberwave/worker-status.json` on the device shows the current phase directly. The file lives on tmpfs, so it costs no flash writes and does not survive a reboot.

### Worker health monitoring

Edge Core continuously monitors the worker container for spontaneous exits and crash loops:

- **Restart accounting**: every restart is recorded with a timestamp and reason.
- **Sliding-window rate limiting**: if more than 5 restarts occur within 5 minutes, the circuit-breaker trips and automatic restarts are suppressed. The breaker resets automatically once the window clears.
- **Spontaneous exit detection**: if the container exits without a deliberate restart, a warning is logged so operators can investigate. The detector recognises deliberate stops via two channels:
  - Callers that hold the monitored `WorkerManager` (e.g. `WorkerManager.stop`) notify it via `record_stop()`. The steady-state workflow deactivation path (`reconcile_worker_lifecycle`) reuses the watcher's monitored manager so this channel covers it directly.
  - For cross-instance stops driven by managers without the shared monitor (the "Restart edge core" flow's `_stop_worker_container_for_restart`, and cold-boot MQTT command handlers that fire before the watcher's first tick), the monitor consults an "expected to be running" probe that returns False when either the workers directory is empty or an edge-core restart is in flight. Both cases downgrade the log to INFO.

  The "may indicate a crash loop" wording is reserved for exits that look unexpected — real crashes, OOM kills, or manual `docker stop` while workflows are active.

Use `cyberwave-edge-core worker health` to inspect the full restart history and circuit-breaker state.

### Resource limits

You can constrain the worker container's CPU and memory usage by setting `CYBERWAVE_WORKER_CPU_QUOTA_PERCENT` and `CYBERWAVE_WORKER_MEMORY_MB` environment variables on the edge host (both optional). When set, Edge Core passes the corresponding `--cpu-quota`, `--cpu-period`, and `--memory` flags to `docker run`.

GPU memory fraction can be limited via `CYBERWAVE_GPU_MEM_FRACTION` (a float between 0 and 1); this is passed as an env var into the worker container.

### High availability and watchdog

Edge Core includes several layers of resilience for edge devices running ML inference workloads (e.g. Raspberry Pi 4 with YOLO). These features prevent the orchestrator from dying under heavy CPU or memory load and ensure automatic recovery when failures occur.

There are **four** independent watchdog/supervision layers, each covering a different failure mode:

| Layer | What it watches | Mechanism | Recovery action |
|---|---|---|---|
| Hardware watchdog | Kernel hang, total board lockup | `/dev/watchdog` ping | SoC resets the board |
| Systemd watchdog | Orchestrator process hang/exit | `Type=notify` + `WatchdogSec` | systemd restarts the service |
| Worker container | ML worker container health | `WorkerHealthMonitor` + circuit breaker | Docker restart / circuit-break |
| Driver containers | Per-driver containers that flap, exit or disappear | `reconcile_driver_*` in `startup.py` | restart suppression on flap; revival of clean exits, and of removed containers after a grace period |

Note: these are *infrastructure* watchdogs. They are **distinct** from the in-process command/teleop watchdogs in `cyberwave-edge-nodes` and `cyberwave-edge-runtime`, which stop driver motion if no commands arrive within a window — the term "watchdog" is overloaded.

#### Systemd watchdog

The systemd unit generated by `cyberwave edge install` configures `WatchdogSec=60` and `Type=notify`. Edge Core pings the systemd watchdog every reconcile cycle (~15 s). If the process hangs or dies, systemd restarts it automatically. The unit uses `Restart=always` (which is the systemd equivalent of Docker's `unless-stopped` — systemd skips the restart when the operator runs `systemctl stop`). `always` is preferred over `on-failure` for robustness: both restart on SIGKILL/OOM, but only `always` also restarts on an accidental `exit(0)` from a buggy code path.

Startup readiness uses a two-phase handoff (`ProcessWatchdog.mark_ready()` → `ProcessWatchdog.start_pinging()`): `READY=1` is sent unconditionally as soon as the blocking boot work (driver images pulled, MQTT connected, twin sync reconciled) is done, while the periodic `WATCHDOG=1` pings — which `sd_notify(3)` restricts to the `$WATCHDOG_PID` process — are driven from the runtime loop in the same process. The split is what makes the standalone PyInstaller binary work under `Type=notify`: the `--onefile` bootloader holds `MainPID` (and therefore `$WATCHDOG_PID`) while the actual Python app runs in a forked child whose PID differs.

During long-running startup operations (driver image pulls and ML worker preparation, including model downloads and the worker image pull), Edge Core sends `EXTEND_TIMEOUT_USEC` notifications via `sd_notify` every 5 seconds. This extends the systemd start timeout while preparation is active, including periods without pull progress events, so `TimeoutStartSec` does not kill the service during a slow download, but not without limit: extensions stop when the startup budget (30 minutes by default) has passed since the watchdog was created, and they stop at once on SIGTERM (see *Bounded startup and shutdown* below). A `STATUS=` notification is also sent so that `systemctl status` shows which images are being pulled. On SIGTERM, startup checks stop advancing to new drivers or workers. An in-flight Docker operation may still finish or reach systemd’s normal stop timeout; it cannot prolong shutdown with startup heartbeats.

Because the watchdog pings come from the runtime loop, anything that blocks the loop for a minute gets the service killed, and an SDK REST call made without a timeout could do exactly that when the backend never answered. Before the runtime loop starts, `main.py` therefore calls `bound_sdk_request_timeouts()`, which patches the generated SDK client so that every request without its own timeout gets one of 10 s (`SDK_REQUEST_TIMEOUT_SECONDS`). Requests that already pass a timeout keep it, the patch is applied once however often it is called, and boot-time calls made before it are not affected. The bound applies to each attempt, not to the call: urllib3's default policy retries a timed-out GET three times, so one GET can still take about 40 s. Nor does it bound a sequence of calls: a driver revival pass makes one GET for the twin list and up to two per linked twin, one after another. The trade-off is deliberate: a legitimately slow REST call from the runtime loop now fails and is retried on a later cycle, instead of taking the whole service down with it.

#### Bounded startup and shutdown

Startup keeps `Type=notify` readiness and a base `TimeoutStartSec=900`. The pull heartbeat above can extend it, but only up to the startup budget, counted from the watchdog's creation, so a startup that never succeeds cannot stay "activating" forever. The budget is 30 minutes; an edge whose first start pulls several gigabytes over a slow link can raise it with `CYBERWAVE_STARTUP_TIMEOUT_MINUTES`, a positive whole number of minutes (any other value is ignored with a warning). Set it in the `envs` of `credentials.json`: the service reads it from there, and so does the unit generator, which derives `JobTimeoutSec` from it and deliberately ignores the shell it runs in, since the service never sees that shell. After changing it, run `sudo cyberwave edge install` again, or let the next package upgrade regenerate the unit, before restarting the service. SIGTERM calls `ProcessWatchdog.begin_shutdown()`: it sends `STOPPING=1` and refuses every later extension, `READY=1` and watchdog ping, even when the signal arrives in the middle of the startup checks, which then exit through the normal shutdown path without announcing readiness. A consequence worth knowing: `/dev/watchdog` is never opened after SIGTERM, so a stop can never arm the hardware reset.

The generated unit also bounds the other end. `TimeoutStopSec=90` gives the orchestrator 90 seconds to stop before systemd kills what is left in the service's control group (`KillMode=control-group`, `SendSIGKILL=yes`), and `JobTimeoutSec` in the `[Unit]` section bounds the start job as a whole, including time spent waiting behind its dependencies, five minutes beyond the startup budget (`35min` by default). Killing the orchestrator does not stop Docker containers by itself, so CLI uninstall checks separately that the service stopped and removes the containers before deleting configuration.

The CLI side matches: on Linux, `cyberwave edge start`, `restart` and `install` submit the systemd job without waiting for readiness and print the commands to follow it, while `stop` and `uninstall` wait up to 120 seconds for the service to stop and, if they cannot confirm it, report the failure and keep the configuration. Deploy both packages to get both halves. The [media and lifecycle investigation](../cyberwave-clis/cyberwave-python-cli/docs/media-device-troubleshooting.md) has recovery steps, limits and verification commands.

#### Hardware watchdog (Raspberry Pi)

On Linux devices with `/dev/watchdog` (all Raspberry Pi models), Edge Core optionally drives the hardware watchdog timer. If the edge-core process stops pinging for longer than the hardware timeout, the SoC resets the board. This is a last-resort safety net for kernel-level hangs or total process death.

Edge Core asks for twice the unit's `WatchdogSec`, and at least 120 seconds, so a hung process is first restarted by systemd and only a hang that outlives that reboots the board. Override it with `CYBERWAVE_HARDWARE_WATCHDOG_TIMEOUT_SECONDS`. A driver may round or clamp the request, and some refuse it outright and keep their own timeout: a Raspberry Pi's `bcm2835_wdt` tops out at 15 seconds. The log line `Hardware watchdog enabled (requested timeout=…, effective=…)` shows the timeout actually running, and a warning follows when it is not above `WatchdogSec`.

Disable with `CYBERWAVE_HARDWARE_WATCHDOG=false`.

#### OOM protection

Edge Core lowers its OOM score (`oom_score_adj=-800`) at startup so the Linux OOM killer preferentially terminates the heavier ML worker container (which Docker restarts automatically via `--restart unless-stopped`) instead of the orchestrator. The systemd unit reinforces this with `OOMScoreAdjust=-800`.

#### Auto-detected memory limits

On devices with ≤4 GB total RAM, Edge Core automatically applies a memory limit to the worker container, reserving ~25% of total memory for the OS and edge-core. This prevents the worker from consuming all available memory and triggering the OOM killer. Explicit `CYBERWAVE_WORKER_MEMORY_MB` values override the auto-detected limit. Disable auto-detection with `CYBERWAVE_WORKER_AUTO_MEMORY_LIMIT=false`.

#### Resource monitoring

`SystemResourceMonitor` (`resource_monitor.py`) watches four host gauges: memory in use, from `/proc/meminfo`; CPU load, the busy share of all cores between two readings of `/proc/stat`; disk usage of the filesystem that holds Docker's data root; and the temperature of the hottest CPU thermal zone. The runtime loop calls `check()` on every tick, but the monitor reads the host at most every 30 s (`MONITOR_CHECK_INTERVAL_SECONDS`) and otherwise hands back its last snapshot, so the CPU figure is a 30 s average and is absent from the first snapshot. Memory at 85 % or a temperature of 75 °C logs `Resource pressure elevated` at INFO. Memory at 92 % or 82 °C logs `CRITICAL resource pressure` as a warning and increments `consecutive_critical`, which drops back to zero on the first reading below both critical thresholds. Each message is rate-limited to one a minute.

CPU, memory and disk also raise an alert when they saturate. `reconcile_resource_saturation` runs after `check()` and looks at each new snapshot once: two snapshots in a row at or above `CYBERWAVE_RESOURCE_SATURATION_PERCENT` (95 % by default; an invalid value logs a warning and falls back to 95) mark the resource saturated, and two in a row below a level five points lower (90 % by default) clear it, so a single spike or dip changes nothing. On the way up Edge Core runs `docker stats --no-stream` once to name the busiest containers, logs `Edge CPU saturated: 97% (threshold 95%) for 30 s; top: cyberwave-driver-658d581b 61%, …`, repeats that line every minute while the resource stays saturated, and sends an `edge_cpu_saturation`, `edge_memory_saturation` or `edge_disk_saturation` warning to every twin whose driver it manages. The alert is raised once per crossing, so the percentage in its description does not defeat the backend's dedupe; its `metadata` carries `resource`, `percent` and `threshold`. A twin the alert could not reach is retried on each later snapshot with the same content until it is delivered or the saturation clears. On the way down the alerts are resolved, and a resolve that fails is retried on the next snapshot. While the backend is unreachable, no alert is raised or resolved: the first call that fails pauses every cloud call of Edge Core, and what is left waits for a snapshot after the pause. The saturation state lives in memory, so a new process cannot know what the last one raised: when the runtime loop starts it treats every resource as possibly alerted, resolves those alerts on every managed twin at the resource's first snapshot below the clear level, and lets a resource that saturates first raise again, which the backend folds into the alert still active. An edge that manages no driver only logs. Temperature keeps its own thresholds and raises nothing.

Board power draw is sampled separately, from sysfs (Jetson INA3221 `in_power0_input`, generic `hwmon/power1_input`, or `power_supply/BAT*/power_now`, in that order), and fresh on every reading rather than from the 30 s snapshot, because draw can change several-fold between idle and motion in under a second. It is published as `power_mw` by the same two transports as the other gauges and shows up as the `pwr X.XW` pill in the Edge details panel of the dashboard.

Set `CYBERWAVE_BATTERY_WH` to your battery pack's watt-hour capacity to enable a rough runtime estimate on the dashboard (e.g. `36.0` for a 36 Wh pack). The estimate is a simple linear model (`battery_wh / current_watts` hours); it ignores discharge curves and temperature effects and is only meant to answer "will this run another 30 minutes?".

| Variable | Default | Description |
|---|---|---|
| `CYBERWAVE_HARDWARE_WATCHDOG` | `true` | Enable/disable the hardware watchdog (`/dev/watchdog`) |
| `CYBERWAVE_HARDWARE_WATCHDOG_TIMEOUT_SECONDS` | twice `WatchdogSec`, at least `120` | Hardware watchdog timeout requested from the driver. Keep it above `WatchdogSec`. See *Hardware watchdog*. |
| `CYBERWAVE_WORKER_AUTO_MEMORY_LIMIT` | `true` | Enable/disable auto-detected memory limits on ≤4 GB hosts |
| `CYBERWAVE_WORKER_STARTUP_PROBE_SECONDS` | `30` | Seconds to wait for the worker container to reach `running` state after `docker run` |
| `CYBERWAVE_WORKER_READY_TIMEOUT_SECONDS` | `900` | Seconds to wait for the worker *process* to finish starting (load and warm up its models) once its container is running |
| `CYBERWAVE_WORKER_STATUS_PATH` | `/run/cyberwave/worker-status.json` | Where the worker writes its startup milestones. Set by Edge Core on the worker container to match the directory it mounts; override only if `/run` is unavailable |
| `CYBERWAVE_BATTERY_WH` | _(unset)_ | Battery pack capacity in watt-hours. Enables the battery uptime estimate in the Edge details panel of the dashboard. |
| `CYBERWAVE_RESOURCE_SATURATION_PERCENT` | `95` | CPU, memory and disk usage at which Edge Core raises the `edge_*_saturation` alerts; they clear five points lower. See *Resource monitoring*. |
| `CYBERWAVE_JETSON_ALLOW_UNSUPPORTED_L4T` | _(unset)_ | Set to `1` to use the `-jetson` worker image and NVIDIA runtime on a Jetson below L4T r38 (JetPack 7). See the accelerator precedence list below. |

### GPU support

When an NVIDIA container runtime is detected (`docker info` reports `nvidia` runtime), Edge Core adds `--gpus all` to the **worker** container's `docker run` command.

For the worker image, Edge Core also rewrites the tag to point at the accelerator-specific sibling. The precedence is:

1. **Jetson host on JetPack 7** (`/etc/nv_tegra_release` reports L4T r38 or newer, or `CYBERWAVE_PLATFORM_VARIANT=jetson`) → `cyberwaveos/edge-ml-worker:<tag>-jetson` + `--runtime nvidia`. Wins over the plain `-gpu` branch: on Jetson the nvidia-container-runtime is present too, but the amd64 `-gpu` blob has the wrong `libcuda.so` ABI (desktop/server CUDA vs Tegra CUDA).
2. **Jetson host on JetPack 6 or older** (L4T r36 and below) → plain `cyberwaveos/edge-ml-worker:<tag>`, no GPU passthrough, with a warning. The `-jetson` image's CUDA 12.9 wheels cannot load against a JetPack 6 driver — cuBLAS fails on the first matmul while convolutions still succeed, so the model loads and then detects nothing. The plain tag is multi-arch, so arm64 CPU inference works on the same host. A `-jetson` tag pinned via `CYBERWAVE_WORKER_IMAGE` is demoted the same way; a pin to an image outside `cyberwaveos/edge-ml-worker` keeps the NVIDIA runtime, since the floor only speaks for our own build. Set `CYBERWAVE_JETSON_ALLOW_UNSUPPORTED_L4T=1` to force the GPU path anyway (e.g. for a hand-built JetPack 6 image).

   The choice is made when the worker container is **created**. A board that already has a `-jetson` worker running keeps it across an Edge Core upgrade — startup skips image selection entirely when the container is already up, and its `--restart unless-stopped` policy survives reboots. Run `docker rm -f cyberwave-worker` once to pick up the new selection.
3. **NVIDIA container runtime detected (not Jetson)** → `cyberwaveos/edge-ml-worker:<tag>-gpu` + `--gpus all`.
4. **`/dev/hailo0` present and no GPU runtime** → `cyberwaveos/edge-ml-worker:<tag>-hailo` + device passthrough.
5. **None** → CPU base image.

If the accelerator-specific tag cannot be pulled (e.g. the sibling isn't published for your channel yet), Edge Core falls back to the base tag and logs the demotion — the worker still comes up, just on CPU.

**Driver GPU passthrough** is opt-in via asset metadata. When a driver's config includes `"prefer_gpu": true` and the host has:

1. NVIDIA container runtime available (`docker info` reports `nvidia`), **and**
2. `nvidia` set as the default runtime in `/etc/docker/daemon.json`

…Edge Core passes `--gpus` to the driver container. The optional `"gpu"` field controls which GPUs are exposed:

| `gpu` value | Docker flag | Use case |
|---|---|---|
| _(not set)_ | `--gpus all` | All available GPUs (default) |
| `1` | `--gpus 1` | Limit to 1 GPU |
| `"device=0,2"` | `--gpus "device=0,2"` | Specific GPU devices |

Example driver metadata:

```json
{
  "drivers": {
    "default": {
      "docker_image": "cyberwaveos/go2-ros2-driver:humble",
      "prefer_gpu": true,
      "gpu": "all"
    }
  }
}
```

If the NVIDIA runtime is available but not configured as the default in `daemon.json`, Edge Core logs an informational message with setup instructions instead of silently skipping GPU passthrough.

### Jetson detection

Edge Core auto-detects NVIDIA Jetson hardware via `/etc/nv_tegra_release`, and reads the L4T major release (`R36` = JetPack 6, `R38` and up = JetPack 7) from its first line. The release gates the ML worker image only — the `jetson-` driver images below build from the same ROS base as their plain counterparts and differ only in a resource profile (smaller pointcloud budget, software H.264, capped publish rates), so they link no CUDA and carry no L4T floor. When running on a Jetson:

- The platform key `linux-aarch64-jetson` is added to the driver resolution order, allowing asset metadata to specify a Jetson-optimised image.
- If no `linux-aarch64-jetson` driver key exists in metadata, Edge Core rewrites the image tag by prepending `jetson-` (e.g. `cyberwaveos/go2-ros2-driver:humble` → `cyberwaveos/go2-ros2-driver:jetson-humble`). If the Jetson-prefixed image is not available, it falls back to the original tag automatically.

Override detection with `CYBERWAVE_PLATFORM_VARIANT=jetson` for testing.

## Multi-camera orchestration

When multiple cameras are connected to the same edge device (each represented as a separate digital twin), Edge Core orchestrates them as follows:

1. **One driver per camera:** Each camera twin gets its own `cyberwave-driver-{uuid[:8]}` container. Child camera twins that are attached to a parent twin share the parent's driver instead.
2. **One shared worker:** A single `cyberwave-worker-{env[:8]}` container receives frames from all cameras. The worker container receives `CYBERWAVE_TWIN_UUIDS` as a comma-separated list of all linked twins.
3. **Readiness probes:** Edge Core waits for all driver containers to reach a `running` state before starting the worker. If some drivers fail, the worker starts anyway so healthy cameras can be utilized.
4. **Model pre-download:** Before the worker starts, Edge Core scans worker scripts and pre-downloads all referenced ML models.
5. **Driver health monitoring:** when a driver container that was running is found stopped or removed, `reconcile_driver_health_for_worker` raises a `driver_health` warning on its twin, whether or not the worker is running.
6. **One device per camera:** `cyberwave edge install` binds each camera twin to its own camera and records the binding under `twin_to_device` in the camera config. For a USB camera the CLI pins a `/dev/v4l/by-id` link, or a `/dev/v4l/by-path` link when two identical units are attached and a by-id name could match either; the value can also be an index (a camera that cannot be pinned), `serial:<serial>` (a depth camera that the driver resolves by serial) or a stream URL. On Linux, Edge Core passes it to the driver as `CYBERWAVE_METADATA_VIDEO_DEVICE` (`_load_selected_camera_device`: an index becomes `/dev/video<N>`, a link or URL goes through verbatim, a `serial:` value injects nothing), and a twin missing from the map gets the global `selected_device` instead. That fallback is why the installer records every pinned twin, even one whose camera the operator chose not to rebind, and every network camera with its stream URL. The injected `/dev` node is also watched, as described below.

   A by-id link resolves inside the container only if the container can see `/dev/v4l`. `_ensure_linux_camera_docker_params` therefore bind-mounts the host's `/dev/v4l` read-only into every image whose name contains `camera-driver` (or `camera_driver`), and into any other image whose final `CYBERWAVE_METADATA_VIDEO_DEVICE`, injected or set in its own params, lies under `/dev/v4l/`, unless the params already mount it.

   Three precedence rules decide what the driver actually receives. `_read_cameras_config` prefers `edge.json`: a non-empty `metadata.cameras` there wins over the legacy `cameras.json` for every key and every twin it lists, and a twin listed only in `cameras.json`'s `twin_to_device` is merged in, so it keeps that entry instead of falling back to `selected_device`. An explicit `-e CYBERWAVE_METADATA_VIDEO_DEVICE=...` in the driver's params wins over the mapping, and Edge Core then injects nothing. And the ALSA audio images, those whose name contains `generic-microphone`, `microphone-driver`, `generic-speaker` or `speaker-driver`, never receive the variable at all.

7. **Network cameras:** a camera on the Ethernet port (a FLIR AX8, an IP camera) has no `/dev` node. `cyberwave edge install` finds it on the wire, identifies it by MAC and writes its stream URL into the twin's metadata (`video_device`, plus `mac_address`) and into `twin_to_device`, so Edge Core hands it to the camera driver like any other source; a URL has no node, so it is not watched for presence. When a driver param still names a source, that param wins as above, which is why the installer removes a placeholder or superseded param from the twin's drivers once it binds the device. How the device is found, matched and pinned, and how to describe a new kind of IP device in asset metadata, is in the CLI's [wired network devices](../cyberwave-clis/cyberwave-python-cli/docs/wired-network-devices.md) guide.

Use `cyberwave edge status` to see all driver and worker containers with their twin mappings.

### Camera pins: drift and missing devices

Two reconcilers in the runtime loop (every ~15 s) keep camera drivers honest about the device they were given.

**Pin drift.** On Linux, `reconcile_camera_config_drift` notices a camera config change by a new modification time on `edge.json` or `cameras.json`. It then compares the `CYBERWAVE_METADATA_VIDEO_DEVICE` each running driver container was started with against the device its twin resolves to now: the twin's own `twin_to_device` entry, or the global `selected_device` when the twin has no entry or its entry resolves to no device (a `serial:` pin, for example). A container started without the variable is skipped, and so is a twin for which neither names a device. When the two differ, for example after `cyberwave edge install --reconfigure-camera`, Edge Core restarts the whole edge on a background thread through `_perform_edge_core_restart`: every driver, the worker and the Zenoh router are stopped and started again, not just the drifted driver. Microphone and speaker containers are never started with `CYBERWAVE_METADATA_VIDEO_DEVICE`, so they cannot trigger that restart, but it recreates them along with everything else.

**Missing devices.** `reconcile_sensor_device_presence` watches each twin that has a `twin_to_device` entry and a driver container started by this Edge Core, and it checks the device the launcher injects for that twin: the entry is resolved by `_load_selected_camera_device`, exactly as at launch, so an index becomes `/dev/video<N>`, a stable link is kept and a `serial:` pin resolves to nothing. Only a result under `/dev/` is checked; serial pins and stream URLs have no node to look for. A node missing on two consecutive checks (about 30 s) raises one `sensor_device_missing` alert on the twin (`error`, `source_type: "edge"`, `metadata.device` naming the path), and two consecutive checks that find it again resolve the alert, so a single glitch neither raises nor resolves one. The alert is also resolved as soon as the twin is no longer checked, because its entry was removed, it was re-pinned to a value with no `/dev` node, or no driver container is mapped to it any more. Camera drivers react to an unplugged device by reconnecting quietly, so without this check nothing told the operator why a camera had gone dark. Microphones are left out on purpose, because the CLI stores their pin in the twin's `audio_device` rather than in `twin_to_device`, and the microphone driver already raises its own sensor-failure alert while its pinned unit is absent. The "already alerted" state lives in memory, so after an Edge Core restart the first check that finds a twin's device present also resolves any alert left from before the restart, once per twin. A resolve that fails, for example while the backend is unreachable, is retried until it succeeds; an alert that fails to send is not retried.

## Multi-container drivers

Some robots require multiple cooperating containers (e.g. a driver, bridge nodes, Nav2, SLAM, and elevation mapping). Edge Core supports this via an optional `services` array in the driver metadata. When present, Edge Core launches one container per service instead of a single driver container.

### Metadata schema

```json
{
  "drivers": {
    "linux-aarch64-jetson": {
      "services": [
        {
          "image": "cyberwaveos/go2-ros2-driver:jetson-humble",
          "name": "driver",
          "command": ["ros2", "launch", "cyberwave_go2_driver", "robot_driver.launch.py"]
        },
        {
          "image": "cyberwaveos/go2-ros2-driver:jetson-humble",
          "name": "bridges",
          "command": ["ros2", "launch", "cyberwave_go2_driver", "robot_bridges.launch.py"]
        },
        {
          "image": "cyberwaveos/ros2-nav2:jetson-humble",
          "name": "nav2"
        },
        {
          "image": "cyberwaveos/ros2-slam:jetson-humble",
          "name": "slam"
        },
        {
          "image": "cyberwaveos/ros2-elevation-mapping:jetson-humble",
          "name": "elevation",
          "prefer_gpu": true
        }
      ],
      "shared_env": {
        "CONFIG_PROFILE": "jetson",
        "ROS_DOMAIN_ID": "0",
        "CYBERWAVE_MAP_DIR": "/data"
      },
      "shared_params": ["--network", "host", "-v", "/data:/data"]
    },
    "default": {
      "docker_image": "cyberwaveos/go2-ros2-driver",
      "prefer_gpu": true
    }
  }
}
```

### How it works

- **`services`** present → multi-container mode (one container per service entry).
- **`docker_image`** present, no `services` → single-container mode (existing behavior, unchanged).
- The `default` fallback key works as before for platforms that don't match a specific key.

### Per-service fields

| Field | Required | Description |
|---|---|---|
| `image` | Yes | Docker image reference |
| `name` | Yes | Service name (used in container naming) |
| `command` | No | Override the container entrypoint command |
| `env` | No | Per-service environment variables |
| `params` | No | Per-service Docker params |
| `prefer_gpu` | No | Enable GPU passthrough for this service |
| `gpu` | No | GPU device selector (default: `all`) |

### Shared configuration

- `shared_env`: Environment variables applied to every service. Per-service `env` overrides shared values.
- `shared_params`: Docker params applied to every service (e.g. `--network host`, volume mounts).
- `service_image_overrides`: for a driver whose image declares its own service graph (no inline `services`), maps service names to image references. Only those services' images change; the declared graph still supplies commands, env, networks, volumes and dependencies. Use it to try one service's build (a mapper, say) on a robot without restating the whole topology. A name the graph does not declare, or a value that is not a non-empty string, stops the driver from starting.

  ```json
  "service_image_overrides": {
    "slam": "cyberwaveos/ros2-slam:jetson-humble-<tag>"
  }
  ```

### Environment layering

1. Edge Core base env (`CYBERWAVE_API_KEY`, MQTT, Zenoh, etc.) — existing, unchanged
2. `shared_env` from metadata
3. Per-service `env` from metadata

### Container naming

- Single-container mode (unchanged): `cyberwave-driver-{twin_uuid[:8]}`
- Multi-container mode: `cyberwave-driver-{twin_uuid[:8]}-{service_name}`

### Backward compatibility

The existing single-image contract is fully preserved. Metadata without a `services` key follows the original code path with zero changes. All existing tests continue to pass unmodified.

## Writing compatible drivers

A Cyberwave driver is a Docker image that interacts with device hardware and the Cyberwave backend. When Edge Core starts a driver container it sets the following environment variables (provided to the container):

- `CYBERWAVE_TWIN_UUID`
- `CYBERWAVE_API_KEY`
- `CYBERWAVE_TWIN_JSON_FILE` (writable file path; set when the twin file exists at launch, see *Twin JSON file*)
- `CYBERWAVE_CHILD_TWIN_UUIDS` (optional, comma-separated)
- `CYBERWAVE_DATA_BACKEND` — data transport backend (`zenoh` by default)
- `ZENOH_SHARED_MEMORY` — `true`/`false`; enables zero-copy Zenoh SHM transport
- `ZENOH_CONNECT` — (optional) comma-separated Zenoh router endpoint URLs
- `CYBERWAVE_METADATA_VIDEO_DEVICE` — (Linux, optional) the camera device this twin is pinned to, from `twin_to_device` or the global `selected_device`. It is never set for microphone or speaker images, and it is left alone when the driver's own params already set it. When it names a `/dev/v4l/` link, the container also gets the host's `/dev/v4l` mounted read-only so the link resolves, whatever the image (see *Multi-camera orchestration*).

Any other `-e CYBERWAVE_METADATA_<KEY>=...` in a driver's params reaches the container as written and, by the convention the Cyberwave camera and microphone drivers follow, outranks the twin's `metadata.<key>`. That makes such a param the strongest way to fix a source, and also the easiest way to leave a stale one behind: a binding written to the twin's metadata is ignored while the param is there. Hardware found on the network is bound through the twin's metadata (the keys its asset's `network_discovery.bind` names, plus `mac_address`), so a driver for a new kind of IP device should read those keys from the twin JSON file below.

`CYBERWAVE_CHILD_TWIN_UUIDS` is present when child camera twins are attached to the driver twin; drivers can use this to coordinate cameras without additional prompts.

### Zenoh data bus

Edge Core automatically injects Zenoh transport configuration into every driver container so that drivers using `cw.data.publish()` work without any extra configuration. The data-bus variables are:

| Variable | Default | Description |
|---|---|---|
| `CYBERWAVE_DATA_BACKEND` | `zenoh` | Data transport: `zenoh` or `filesystem` |
| `ZENOH_SHARED_MEMORY` | `false` | Opt-in zero-copy shared-memory transport. Requires `--ipc=host` between containers; leave disabled unless your runtime is configured for it. |
| `ZENOH_CONNECT` | (empty) | Router endpoints, e.g. `tcp/10.0.0.1:7447` |

All variables can be overridden per-driver with `-e KEY=VALUE` in driver params.

**Peer-to-peer mode (default):** when `ZENOH_CONNECT` is empty, Zenoh uses multicast discovery. On Linux with `--network host` (the default), all driver containers on the same machine discover each other automatically.

**Router mode (optional):** set `ZENOH_ROUTER_ENABLED=true` to have Edge Core start an `eclipse/zenoh:1.9.0` router container before the driver containers. This is required for MQTT bridge or multi-hop topologies.

Environment variables for Zenoh infrastructure:

| Variable | Default | Description |
|---|---|---|
| `ZENOH_ROUTER_ENABLED` | `false` | Start a Zenoh router container before drivers |
| `ZENOH_ROUTER_IMAGE` | `eclipse/zenoh:1.9.0` | Docker image for the router |
| `ZENOH_ROUTER_PORT` | `7447` | Host port for the router |
| `ZENOH_SHARED_MEMORY` | `false` | Opt-in shared-memory transport. Requires all Cyberwave containers to share an IPC namespace (`--ipc=host`); leave disabled unless validated end-to-end. |

### Host telemetry

Edge Core reports the host's resource pressure over two transports. The bootstrap publisher folds the gauges into every `cyberwave/twin/{uuid}/edge_health` heartbeat it sends (~5 s), but Edge Core stops that publisher once driver startup has brought up at least one driver, because drivers publish their own health on the same topics from then on. After that the gauges travel only on the REST host-facts keepalive, which uploads them every ~30 s together with the static hardware facts. In practice, then, MQTT carries host metrics while the edge boots (and beyond that only on an edge whose startup brought up no driver), and REST carries them the rest of the time.

**Dynamic (MQTT `edge_health` payload, every ~5 s while the bootstrap publisher runs):**

| Field | Source | Notes |
| --- | --- | --- |
| `host_memory_percent` | `/proc/meminfo` | Absent on non-Linux hosts. From the monitor's 30 s snapshot. |
| `host_memory_available_mb` | `/proc/meminfo` | Free + buffers/cache. From the monitor's 30 s snapshot. |
| `cpu_temp_c` | `/sys/class/thermal/thermal_zone*` | Hottest CPU-typed zone. From the monitor's 30 s snapshot. |
| `host_cpu_percent` | `/proc/stat` | Busy share of all cores between the monitor's last two snapshots. Absent on non-Linux hosts and before the second snapshot. |
| `host_disk_percent` | `shutil.disk_usage` | Usage of the filesystem holding Docker's data root. From the monitor's 30 s snapshot. |
| `consecutive_critical` | `SystemResourceMonitor` | Consecutive 30 s readings over a critical threshold (memory 92 %, CPU 82 °C). Resets on the first reading below both. |
| `power_mw` | sysfs power rails | Read fresh on every heartbeat; see *Resource monitoring*. |
| `battery_wh` | `CYBERWAVE_BATTERY_WH` | Only when set. |
| `watchdog_layers` | `ProcessWatchdog.active_layers()` | E.g. `["systemd"]` or `["systemd", "hardware"]`. Also on `host_facts`, which is where the dashboard reads it — see below. Still published here so a dashboard can keep showing layers for an edge whose binary predates that upload. |

Only the bootstrap publisher on the edge host injects these fields. Driver containers see their container's `/proc`, not the host's, so they intentionally omit the host block and publish only stream-level health.

**Per-stream config:**

Each entry under `streams[id]` may carry an optional `stream_config` block — a discriminated union keyed by `kind` (`camera` / `audio` / `lidar` / `imu`) describing the runtime-negotiated config for that stream. Drivers attach a block once at startup via `EdgeHealthCheck.register_stream_config(stream_id, config)` in the Python SDK; the publisher merges it into every heartbeat thereafter. The legacy top-level `camera_config` slot stays populated for one deprecation release for out-of-tree consumers and mirrors the lexicographically-first camera-kind `stream_config`. New consumers must read `streams[id].stream_config` instead — the legacy slot only ever describes one camera and is silent for audio / lidar / IMU streams.

`reconcile_camera_config_drift` works from file modification times and `docker inspect` output and never reads an `edge_health` payload, so changes to this schema cannot affect it (see *Camera pins: drift and missing devices*).

**Static (REST `POST /api/v1/edges/discover`, every ~30 s):**

Edge Core uploads a `host_facts` object — total RAM, CPU model and core count, kernel, thermal source, `/dev/watchdog` availability and the active watchdog layers, non-loopback network interfaces, a primary MAC address, the board serial number, and the current CPU-temperature / memory / power gauges (plus `battery_wh` when set) — into `Edge.metadata['host_facts']`. Those gauges duplicate the heartbeat fields above on purpose: the bootstrap publisher stops at the first driver start, and a host row that lost a reading there looked like a fault rather than a transport change. The first POST is performed by a background daemon thread as soon as it starts, then the same thread re-reads and re-POSTs every ~30 s (a fresh read each tick, not a cached payload — the interface list and the gauges below depend on that). Running the bootstrap POST off the main thread is deliberate: under systemd `Type=notify`, the service signals `READY=1` from the main thread, and we don't want a slow or unreachable backend to push past `TimeoutStartSec` (which would cause `systemctl restart cyberwave-edge-core` to fail). The call is idempotent (it upserts `Edge.metadata['host_facts']` and bumps `Edge.last_seen_at`), so the keepalive cost is one tiny round-trip per edge regardless of how many bound twins the edge has — or whether it has any at all.

`network_interfaces` — each interface's name, IPv4 address, MAC and up/down state — exists so an edge that reconnects to a different network (e.g. a different WiFi AP) can still be found and SSH'd into without scanning the LAN. Because it's the one field here that legitimately changes over the device's lifetime, Edge Core additionally publishes a one-line `driver_log` message (`"Network interfaces: eth0=192.168.1.42 (up), ..."`) to every twin bound to the edge whenever the observed interface set changes, so the change shows up in that twin's Logs tab with a timestamp — not just as a silently-overwritten snapshot.

Three properties of that fan-out are load-bearing:

- **Dedup is per twin, not per process.** The set of bound twins grows over an edge's lifetime, so "already published" has to be a per-twin fact — a twin bound after the first publish has never seen the line and still gets one baseline message. Signatures live in memory only, so a restart re-publishes one baseline line per twin.
- **A twin's signature is recorded only after its message reached a connected broker.** The publish path swallows failures, so recording up-front would mean a broker that happened to be unreachable on the tick where the IP changed loses that line permanently. Nothing is recorded on failure, so every later tick retries until it lands.
- **The fan-out runs on its own daemon thread, never on the keepalive thread.** An MQTT connect against an unreachable broker can burn ~50 s (3 attempts × CONNACK timeout + backoff), which inline would stall the next `/discover` POST and eat the margin on `Edge.last_seen_at` — a liveness signal whose whole point is to be independent of MQTT. Only one attempt is ever in flight; a tick that finds one running skips and retries later.

Interfaces with no IPv4 address bound are excluded from both the message and the change signature. They're nothing anyone can SSH to, and they churn (`docker0`/`veth*` as driver containers cycle, `usb0` as cables move), which would otherwise re-fire the fan-out on changes that tell an operator nothing.

The Edges tab reaches the same goal by a different route: it renders only the interface named by `primary_interface_name` and keeps the full inventory on hover, so an unplugged NIC's MAC (what you pin a DHCP reservation to) and a VPN interface (how you reach a remote edge) stay one hover away without either crowding out the routable address. The exclusion here is specifically about log-line churn.

That `Edge.last_seen_at` field is what powers the three-state dashboard pill:

| State | Trigger | Meaning |
| --- | --- | --- |
| **Online** | `Edge.last_heartbeat` fresh (≤60 s) | A bound twin is publishing MQTT `edge_health`. |
| **Standby** | `Edge.last_seen_at` fresh (≤90 s) but no recent MQTT | Edge Core daemon is alive and posting REST keepalives; no twin is doing work. |
| **Offline** | Neither signal within window | Process is down, or partitioned from both transports. |

The 30 s cadence is deliberately tighter than the 90 s Standby window so a single missed POST (slow DNS, transient TLS handshake) doesn't drop the row to Offline — three missed POSTs do. Failure of any individual upload logs at `debug` and falls through to the next tick.

Coverage per platform:

| Field | Linux | macOS | Notes |
| --- | --- | --- | --- |
| `platform`, `kernel` | yes | yes | From `platform.platform()` / `platform.release()`. |
| `memory_total_mb` | `/proc/meminfo` | `sysctl hw.memsize` | Rounded to one decimal on both platforms. |
| `cpu_model` | `/proc/cpuinfo` | `sysctl machdep.cpu.brand_string` | E.g. `"Apple M2 Pro"`. |
| `cpu_count` | `/proc/cpuinfo` (`processor:` records) | `sysctl hw.logicalcpu` → `hw.ncpu` | Logical CPU count on both platforms (after SMT expansion). |
| `thermal_source` | first CPU thermal zone | omitted | Tracks the path the live publisher reads from. Darwin has no thermal reader today, so the field stays absent rather than claim a source that isn't sampled. |
| `has_hardware_watchdog` | `/dev/watchdog` | always `false` | Linux-only kernel capability. Says the device node exists, not that anything is pinging it — prefer `watchdog_layers`. |
| `watchdog_layers` | `ProcessWatchdog.active_layers()`, overlaid by Edge Core | same | Which layers are actually supervising this process, in protocol order. The one key here the SDK's `read_host_facts()` does not produce: it probes `/dev/watchdog` and so cannot see systemd. Omitted, never `[]`, when no layer is enabled, so "unprotected" and "edge predating this upload" stay distinguishable — the dashboard falls back to `has_hardware_watchdog` only for the second. Note that `"hardware"` is absent from the *bootstrap* POST even on a board that has the device: `HardwareWatchdog.enabled` only flips inside `open()`, which `main.py` calls after `run_startup_checks()` returns, so the layer appears one keepalive tick later. The dashboard renders that gap — and a permanently failing `open()` — as `watchdog: systemd (+hw idle)`. Rides REST rather than the heartbeat for the same reason the gauges below do. |
| `network_interfaces` | `/sys/class/net/*` (name, `address`, `operstate`; IPv4 via `SIOCGIFADDR`) | `ifconfig -a` (name, `ether`, `status:`, `inet`) | The full inventory, deliberately unfiltered beyond loopback — a VPN interface (`wg0`, `tailscale0`) is often how you reach a remote edge, so this is not the place to decide what's "reachable". `ipv4_address` is `null` when the interface has no bound address (e.g. down); `mac_address` stays populated, since it's what a DHCP reservation keys on. On Linux, expect it to be long: an edge running driver containers reports 20+ entries (`docker0`, a `br-*` per compose network, a `veth*` per container, kernel tunnel pseudo-devices). Consumers should render `primary_interface_name` and keep the rest behind a disclosure. macOS narrows ~30 interfaces to the real NICs using three signals read out of the `ifconfig` text: a name prefix for the families Apple names consistently (`utun*`, `awdl*`, `bridge*`, `anpi*`, …); `media: none`, which means no physical layer at all and catches the internal `anpi` peers that are named `en6`–`en9`; and membership of a `bridge*`, which catches the Thunderbolt ports `en2`–`en5` (the bridge stanza enumerates its own members, so their names are not assumed). An *unplugged* real NIC reports `media: autoselect (none)` — a media layer with no link — and is deliberately kept, since its MAC is what a DHCP reservation keys on. Measured on an M-series Mac: 32 interfaces → 3 (`en0`, `en1`, and one unplugged port), payload 587 B. Link state comes from `status:` rather than the `RUNNING` flag, which Apple's internal NICs assert permanently. |
| `primary_interface_name` | interface holding the default route | interface holding the default route | Name of the one interface the host actually routes out of, resolved by asking the kernel for the source address it would use (a UDP `connect()` to TEST-NET-1 — no packet is transmitted), then matching that address back to the enumerated set. Structural rather than heuristic: a bridge or `veth` peer cannot hold the default route, so no name blocklist is needed. Falls back to a hardware-backed interface (`/sys/class/net/<if>/device` exists) that is up when the host has no route at all. Omitted when the MAC came from the `uuid.getnode()` last resort. |
| `primary_mac_address` | MAC of `primary_interface_name` → `uuid.getnode()` | MAC of `primary_interface_name` → `uuid.getnode()` | **The** MAC — one value, resolved the same way on Linux, macOS, Jetson and Raspberry Pi, and the one to put in a DHCP reservation. Consumers need no selection logic of their own; picking client-side from `network_interfaces` by enumeration order lands on `docker0` (sysfs enumeration is alphabetical and container bridges are up). Falls back to `uuid.getnode()` when nothing was enumerated — not tied to a reachable interface, but the only MAC available there; `primary_interface_name` is then omitted so the two provenances stay distinguishable. Omitted entirely when `getnode()` fabricated a random node ID (RFC 4122 flags this with the multicast bit), since a MAC-shaped string matching no NIC is worse than nothing. |
| `device_serial` | `Serial` in `/proc/cpuinfo` (Raspberry Pi) → `/sys/firmware/devicetree/base/serial-number` (Jetson) | omitted | Board serial for asset tracking / RMA. Device-tree values are NUL-terminated and stripped. All-zero placeholder serials (`0000000000000000`) are treated as absent. |
| `cpu_temp_c` | hottest CPU thermal zone | omitted | Live gauge, not a fact — see below. Deliberately the *hottest* zone, while `thermal_source` names the *first*; the two disagree by design. |
| `memory_used_percent`, `memory_available_mb` | `/proc/meminfo` | omitted | Live gauges — see below. `sysctl` gives Darwin the total but not the free/available split, so both stay absent there. |
| `sdk_version`, `edge_core_version` | in-process `__version__` → `importlib.metadata` | in-process `__version__` → `importlib.metadata` | Version of the SDK and Edge Core **actually loaded** by the running daemon. The in-process lookup preserves the `BUILD_VERSION` stamp injected by CI for `.deb` builds and pre-release wheels, and survives PyInstaller binaries that ship without `.dist-info`, so the Edges tab agrees with `cyberwave-edge-core --version`. CLI version is intentionally not part of `host_facts` — the CLI ships as a separate PyInstaller binary, so Edge Core cannot observe it from its own Python process; `cyberwave --version` remains the source of truth for the CLI an operator is running locally. |

#### Why two gauges ride a facts payload

`cpu_temp_c` and the `memory_*` gauges are the same readings the MQTT `edge_health` heartbeat publishes every ~5 s, duplicated onto the ~30 s REST keepalive. The duplication is not redundancy for its own sake — without it those readings have no route off a working edge at all:

- The bootstrap `edge_health` publisher is the only thing that samples them, and Edge Core **stops** it as soon as the first driver starts (drivers own the twin's health topic from then on).
- A containerised driver sees its own `/proc` and has no access to the host's thermal sysfs, so it cannot take over the sampling.

The net effect before this existed: memory and temperature appeared while an edge sat idle and vanished the moment it started doing work — exactly backwards from when an operator wants them. Consumers that may see both copies should prefer whichever is *fresher*, which is the heartbeat's only while heartbeats are still arriving — a consumer holding the last-received payload in memory keeps those numbers indefinitely, so once the heartbeat stops they become a frozen snapshot and this copy is the live one. The dashboard treats a stale heartbeat as having no gauges at all and falls through to these. Treat a fallback reading as up to 30 s stale at the source, plus whatever polling interval the consumer adds on top.

### Local pages and the edge local portal (opt-in)

A driver that serves a page of its own on the edge's network (a download page
for a ground-station browser, a camera preview, a status page) declares it in
its `cw-driver.yml`:

```yaml
local_http:                 # shorthand for drivers.default.local_http
  - name: plans             # slug, unique per driver
    port: 8765              # the port the driver itself listens on
    path: /                 # absolute path; no query string or fragment
    service: gateway        # optional: which service of a multi-container driver serves it
    audience: lan           # lan (default) is listed; host is declared but never listed
    scheme: http            # http (default) or https
    description: Plans for the ground station
```

The backend validates the block and stores it on the driver profile
(`drivers.<profile>.local_http`); a twin can override it through its own
`metadata.drivers`, with the same precedence Edge Core uses to pick the image.
The driver still opens its own port: the declaration only makes it
discoverable.

With the portal switched on, Edge Core serves a small read-only index of those
pages:

| Variable | Default | Meaning |
|---|---|---|
| `CYBERWAVE_EDGE_LOCAL_PORTAL` | off | `on` / `true` / `1` starts the portal. Anything else leaves Edge Core with no HTTP surface. |
| `CYBERWAVE_EDGE_LOCAL_PORTAL_BIND` | `0.0.0.0:8700` | `host:port`, `[v6]:port`, `:port` or `port`. Use `127.0.0.1:8700` to keep it on the host. |
| `CYBERWAVE_EDGE_LOCAL_PORTAL_HOST` | address the client reached | Host name or IP printed in links, e.g. a DNS name your LAN resolves. |
| `CYBERWAVE_EDGE_LOCAL_PORTAL_REFRESH_SECONDS` | `10` | How often the listing is rebuilt from `docker ps` and the twin files. |

Set them in the Edge Core service environment or under `envs` in
`credentials.json`, then restart Edge Core.

- `GET /` lists, per running driver, its twin name, registry id, health (from
  the container healthcheck) and a link to `http://<edge-ip>:<port><path>` for
  each declared page. A page whose `service` container is not running is shown
  as `stopped`, without a link.
- `GET /portal.json` is the same listing, machine-readable
  (`schema: cyberwave.edge.local_portal.v1`).
- `GET /healthz` reports the portal's own liveness.

What it does not do, by design:

- **No proxying.** Links point straight at the driver's port, so the portal
  adds an index and nothing else to the edge's attack surface. Protecting a
  page (if it needs it) is the driver's job.
- **No secrets.** The output is built from a fixed set of fields. Tokens,
  `shared_env`, image names and the rest of the twin metadata never reach it,
  and declared paths cannot carry a query string. Entries are re-validated on
  the edge, because twin metadata is editable after the catalog checked it.
- **No mDNS.** Edge Core has no mDNS dependency and this does not add one;
  reach the portal by IP or by a name your network already resolves.
- **Never blocks the runtime loop.** Docker and the twin files are read by a
  background refresher; requests are answered from that snapshot. A bad bind
  value or a busy port is logged and Edge Core runs on without the portal.

The links use the address the client connected to (or
`CYBERWAVE_EDGE_LOCAL_PORTAL_HOST`), never the request's `Host` header.

### Driver failure handling

Drivers should exit with a **non-zero** code when they cannot access required hardware at startup (for example, missing `/dev/video*` or disconnected peripherals). This allows Edge Core to detect startup failures and trigger restart logic. Not every driver does: the camera driver keeps running and reconnects when its device is unplugged, which is why Edge Core watches camera pins itself (`sensor_device_missing` below).

Edge Core raises these twin alerts, all with `source_type: "edge"`:

| Alert type | Severity | Raised when | Cleared by Edge Core |
|---|---|---|---|
| `driver_starting` | `info` | A driver start attempt begins; carries pull and startup progress (see *Remote restart*). | Resolved once the container runs. A failed attempt is annotated with its phase, escalated to `critical` and resolved. |
| `worker_starting` | `info` | The same for the ML worker, held until the worker reports ready (see *Worker readiness*). | As above. |
| `driver_start_failure` | `error` | The twin has no asset, the driver image cannot be pulled, or the container does not reach a running state. A launch refused only by its restart floor raises nothing. | Resolved by the next successful start of that twin's driver. |
| `driver_restart_loop` | `error` | The driver restarted more than `CYBERWAVE_DRIVER_RESTART_LOOP_THRESHOLD` times within `CYBERWAVE_DRIVER_RESTART_LOOP_WINDOW_SECONDS`. The container is switched to `--restart=no` and stopped. | No. |
| `driver_health` | `warning` | A driver container that was running is found stopped or removed. | No. |
| `sensor_device_missing` | `error` | The `/dev` node a managed camera twin is pinned to is missing on two consecutive checks (see *Camera pins: drift and missing devices*). | Resolved when two consecutive checks find it again, or when the twin is no longer checked. |
| `driver_restart_needed` | `warning` | A configuration key in the metadata of a twin whose driver is not a sensor driver, or declares a restart floor, changed (see *Bidirectional twin sync*). Edge Core restarts nothing. | No. |
| `driver_metadata_restart_suppressed` | `warning` | A sensor driver has already been restarted three times in 30 minutes for metadata changes; the change is left for a manual restart. | No. |
| `edge_cpu_saturation`, `edge_memory_saturation`, `edge_disk_saturation` | `warning` | Raised on every managed twin when CPU, memory or disk usage is at or above `CYBERWAVE_RESOURCE_SATURATION_PERCENT` on two consecutive 30 s snapshots (see *Resource monitoring*). | Resolved after two consecutive snapshots five points below the threshold. |
| `model_download_failure` | `error` | A required ML model cannot be downloaded. | No. |
| `worker_start_failure` | `error` | The ML worker container cannot be started, or its startup probe gives up (see *Worker startup probe*). | Resolved by the next successful worker start. |
| `workflow_sync_failure` | `error` | Downloading a twin's workflow workers from the backend failed. | No. |
| `macos_camera_not_configured`, `macos_microphone_not_configured`, `macos_speaker_not_configured` | `warning` | On macOS, a camera, microphone or speaker twin has no host stream or bridge URL configured. | No. |

When a model fails to download, the worker container is **not** started: a workflow that depends on the model is held off rather than allowed to run with empty inference results. Models that the worker container can resolve on its own (an Ultralytics auto-download on a frozen PyInstaller binary, for example) are deferred via `_RuntimeManagedDeferred` and never trip the alert.

`docker_create_timeout` is a failure phase, not an alert type. When `docker create` times out and the container has still not appeared 10 s later (see *When `docker create` times out*), the `driver_starting` alert is closed with `phase: "docker_create_timeout"` and the twin gets a `driver_start_failure`. Nor does Edge Core create the `edge_core_restart` alert: the backend creates it when a restart is requested through `POST /api/v1/edges/{uuid}/restart-core` and ships its UUID in the MQTT command, and Edge Core only moves its `metadata.phase` from `in_progress` to `completed` or `failed` (`EdgeCoreRestartAlertContext` in `utils.py`).

Optional environment variables to tune restart behavior:

- `CYBERWAVE_DRIVER_RESTART_LOOP_THRESHOLD` (default: `4`)
- `CYBERWAVE_DRIVER_RESTART_LOOP_WINDOW_SECONDS` (default: `60`)
- `CYBERWAVE_DRIVER_CREATE_TIMEOUT_SECONDS` (default: `30`)
- `CYBERWAVE_DRIVER_REMOVED_REVIVAL_SECONDS` (default: `300`; `0` keeps removed drivers down)
- `CYBERWAVE_DRIVER_TROUBLESHOOTING_URL` (default: `https://docs.cyberwave.com`)

#### Periodic Docker cleanup

Edge Core prunes stopped `cyberwave-*` containers every 30 minutes, and unused images and anonymous volumes every 3 hours, from the runtime loop (`_run_periodic_docker_cleanup`). When Docker's data root is on an SD card the pruning is deferred to spare the flash, but only until that filesystem is 80 % full; `CYBERWAVE_SKIP_PERIODIC_DOCKER_CLEANUP=1` turns it off altogether. The intervals, the threshold and the reasoning are under *Disk usage: container logs and cleanup*.

#### Driver revival and orphan containers

When a managed driver exits cleanly (Docker's `--restart unless-stopped` policy does not auto-revive clean exits), Edge Core's revival reconciler re-runs driver startup for that container so it is recreated. Revival is restricted to driver containers this Edge Core process launched. A service declared `restart: "no"`, or one the flap detector stopped, is left down, unless its relaunch was refused only on its restart floor. If `docker ps` fails or times out, the health snapshot from the previous cycle is kept and nothing is classified, alerted or revived on the strength of the failed listing.

A managed container that has been **removed** altogether, by `docker rm` or by the periodic prune, is revived too, but only after a grace period: 300 s by default, set with `CYBERWAVE_DRIVER_REMOVED_REVIVAL_SECONDS`, where `0` disables it. The grace leaves an operator time for a deliberate takedown, an image swap or a debugging session before Edge Core brings the driver back. `_removed_containers_due` notes when each container was first seen removed and hands the overdue ones to the same revival pass as clean exits, so the orphan filter, the flap detector's opt-out and the 60 s revival debounce all apply. A removed container cannot be inspected, so Edge Core records the restart policy it launched each container with, and marks the ones the flap detector stopped as `no`: a removed `restart: "no"` or flap-stopped driver is not revived, unless its relaunch was refused on its restart floor. If a revival pass finds no linked twin that still declares a removed container, the container is not forgotten at once: that takes three passes in a row without a declaring twin (`_REMOVED_FORGET_AFTER_MISSES`), each at least a minute apart, so a twin list that comes back short while the backend is flaky does not cost a driver, and a twin that is really gone costs three passes rather than one a minute. Containers removed by an edge restart are dropped from Edge Core's bookkeeping and never come back this way; the restart relaunches what it still needs.

Stopped `cyberwave-driver-*` containers that this process did not start, such as those of twins unlinked before Edge Core last started, are treated as **orphans** and ignored by revival. The periodic prune removes them with every other stopped `cyberwave-*` container, except those that carry a restart-floor label, or when periodic cleanup is disabled. Without this guard, an orphan would send a revival pass to the backend every minute for a container that no linked twin declares.

### Twin JSON file

`CYBERWAVE_TWIN_JSON_FILE` points the driver at `/app/<twin_uuid>.json`, a JSON copy of its twin: the twin instance as the API returns it (TwinSchema, including `metadata`), with the catalog asset (AssetSchema) under `asset`. Edge Core sets the variable only when the host file exists at launch.

On the host the file is `<config_dir>/<twin_uuid>.json`, and the container sees it twice: as a writable single-file bind mount at `/app/<twin_uuid>.json`, and inside the read-only mount of the whole config directory at `/app/.cyberwave/<twin_uuid>.json`. The difference matters because a single-file bind mount follows the file's inode, not its path. That is why `write_or_update_twin_json_file`, which runs before every driver launch, deep-merges the backend twin into an existing file and rewrites it in place rather than replacing it: the inode a running container holds stays the one the host sees.

Drivers read the file when they start. The Cyberwave camera driver's `entrypoint.sh`, for example, exports the twin's metadata as `CYBERWAVE_METADATA_*` variables once, before the driver process runs, so a change to the file while the driver runs does not reach the driver until it is started again.

#### Bidirectional twin sync

`reconcile_twin_json_file_sync` runs on every runtime tick (~15 s) and keeps the host files and the backend in step in both directions.

The push leg looks for files whose SHA-256 changed since the previous tick and sends each one with `PUT /api/v1/twins/{uuid}`. The payload is every field of `_TWIN_UPDATE_ALLOWED_FIELDS` present in the file: the whole `metadata` object along with the name, pose, attachment and calibration fields, but never asset or environment assignments. This is how a driver's own edits to its twin file reach the backend. A file seen for the first time only has its checksum recorded, and the writes Edge Core makes itself, at launch and on a pull, record the new checksum too, so they are never pushed back.

The pull leg covers every file the push leg did not send in that tick. `_pull_twin_json_file_from_backend` fetches the twin with `client.twins.get_raw` and, for each field in `_TWIN_PULL_ALLOWED_FIELDS` (only `metadata`), compares the backend value with the local one as a whole; if they differ at all, the backend value replaces the local one outright, with no key-by-key merge. A push that failed therefore does not protect a local edit, because the same tick's pull can overwrite it with the backend's metadata. For the same reason a field the edge legitimately writes must never be added to `_TWIN_PULL_ALLOWED_FIELDS`.

Like the launch path, the pull rewrites the host file in place, so the inode a running container holds stays the one the host sees: `/app/<twin_uuid>.json` shows the pulled content at once, and what the driver writes there still reaches the push leg. The new content is first written to a temporary file, so a full disk fails before the original is touched, and the original is then overwritten without being emptied first, a shorter document padded with spaces until the old tail is cut, so a reader always finds valid JSON. A host file that cannot be read is rebuilt from the last copy Edge Core wrote or read intact, with the backend metadata applied; with no such copy in memory it is left, with one warning, for the next launch to rewrite. The pull reports which top-level metadata keys changed, and `reconcile_twin_json_file_sync` returns them per twin.

Drivers read the file only when they start, so a metadata edit made in the UI, such as a new microphone gain, reaches a driver only through a restart. `reconcile_driver_metadata_restarts`, which runs right after the sync, makes that restart for sensor drivers: the camera, microphone, speaker and webcam drivers, recognised by their image names.

- **Configuration only.** Runtime state is left out of the comparison: telemetry (`status`, `power`, `motor_temps`, `performance`), the backend's navigation keys (`active_map_*`, `finalize_active_map_request`, `last_finalized_map_*`, `navigation_lifecycle`, `localization`), the drivers' own reports (`speaker_device`, `audio_routing`, `robot_dump`, and `audio_device` when it is the webcam's dict rather than a pinned device string) and `edge_fingerprint`; a change of `edge_fingerprint`, a link or unlink, never restarts anything. In `sensors`, entries without `parameters` are left out too, because a driver registers a missing sensor on its own as a bare id and type, and a list left empty counts as no `sensors` at all. Every other key is configuration, compared as a digest of the filtered metadata.
- **Debounced.** The restart happens once the filtered metadata has been unchanged for at least 10 s, that is on a later tick, so several saves in a row cost one restart.
- **The driver's own writes do not count.** The digest written at launch is the baseline, and whatever Edge Core pushes from the file becomes the new one; the writes a driver is known to make through the backend are filtered out of the digest (see above). Each of the twin's containers is judged on its own: one that started after the change, or is not running, has it or reads it when it starts. A change seen during a container's first 90 s waits for those 90 s to pass, so nothing restarts while a driver is starting; if the filtered metadata still differs from what the containers started with, the twin is then restarted. A real edit made right after a restart therefore still reaches the driver, and a startup write Edge Core does not recognise costs at most the restarts the rate limit allows. A container that cannot be inspected holds the decision until a later tick.
- **Relaunch.** The twin's containers go through `fetch_and_run_twin_drivers(..., only_container_names=..., pull_images=False)`, the revival path, which re-fetches the twin, rewrites the file and shows `driver_starting`. A child camera twin restarts the containers of the twin it is attached to. The relaunch uses the image already on disk and pulls only a missing one: a settings change applies settings and does not upgrade the driver. It also runs inside the runtime loop, whose only `WATCHDOG=1` comes at the end of each cycle, so a pull of a newly published tag could outlast `WatchdogSec=60` and get Edge Core aborted. For the same reason the relaunch gets the loop's watchdog, which the dependency health wait pings. The change counts as applied only when every relaunched container started; otherwise it stays pending and is retried under the rate limit below.
- **Rate limit.** At least 120 s between two metadata restarts of a twin and at most three in 30 minutes; beyond that the change is left for a manual restart and the twin gets one `driver_metadata_restart_suppressed` alert.
- **Never during another restart.** The step waits while an Edge Core or camera-drift restart is in progress.

Any other driver, a robot or a drone, and any service with a restart floor, is not restarted for a metadata change: the change raises one `driver_restart_needed` alert ("Settings changed; restart the driver to apply them") on the twin whose settings changed. Until that alert, or a `driver_metadata_restart_suppressed` one, reaches the backend, the change stays pending and the alert is retried on later ticks.

### Twin metadata

Use the official Cyberwave SDK to interact with the API and MQTT; it abstracts authentication, retries, and handshake logic.

Register a driver by adding its configuration to a twin's metadata (or the catalog twin's metadata if you control the catalog twin). Use the environment view's **Advanced editing** to edit metadata.

Note: changing a catalog twin's metadata affects all subsequently created digital twins derived from that catalog twin.

Example driver metadata (JSON):

```json
{
  "drivers": {
    "default": {
      "docker_image": "cyberwaveos/so101-driver",
      "version": "0.0.1",
      "params": [
        "--network",
        "local",
        "--add-host",
        "host.docker.internal:host-gateway"
      ]
    }
  }
}
```

### Platform-specific driver selection

Edge Core can select platform-specific driver entries before falling back to
`default`.

Selection order:

1. Child-registry-specific entry (existing behavior)
2. Host platform/machine keys (for example `darwin-arm64`, `darwin`, `macos`, `mac`)
3. `default`

Example:

```json
{
  "drivers": {
    "default": {
      "docker_image": "cyberwaveos/so101-driver"
    },
    "darwin-arm64": {
      "docker_image": "cyberwaveos/so101-driver:macos",
      "params": ["-e", "CYBERWAVE_SERIAL_BRIDGE_URL=tcp://host.docker.internal:22001"]
    }
  }
}
```

### macOS host-device bridge hook

On macOS, Linux `--device` mappings in `params` cannot directly expose host
hardware to Linux containers. Edge Core now supports a pre-run native bridge
hook:

- Set `CYBERWAVE_MACOS_DEVICE_BRIDGE_COMMAND` on the host
- Edge Core executes it once per `--device` mapping before `docker run`
- Template variables available:
  - `{host_device}`
  - `{container_device}`
  - `{twin_uuid}`
  - `{container_name}`
  - `{config_dir}`

Example:

```bash
export CYBERWAVE_MACOS_DEVICE_BRIDGE_COMMAND="cyberwave-edge-hw-bridge --device {host_device} --target {container_device} --twin {twin_uuid}"
```

The command can start native camera/serial forwarding services that expose
bridge endpoints to the container (typically via `host.docker.internal`).

Bridge command stdout can optionally return a resolved source for the mapped
device:

- JSON: `{"resolved_device":"rtsp://host.docker.internal:8554/cam0"}`
- or line format: `resolved_device=rtsp://host.docker.internal:8554/cam0`

When this value differs from `/dev/video*`, Edge Core can transparently:

- inject `CYBERWAVE_METADATA_VIDEO_DEVICE` for the driver
- inject `CYBERWAVE_CHILD_CAMERA_STREAM_URLS` with the per-child MJPEG mapping
  for multi-camera parent drivers such as SO101
- inject `CYBERWAVE_EDGE_VIDEO_DEVICE_MAP` (JSON map of Linux device to resolved source)
- remove Linux-only `--device /dev/video*` flags before `docker run` on macOS
  (default enabled with `CYBERWAVE_MACOS_STRIP_VIDEO_DEVICE_PARAMS=true`)

This lets Linux-style drivers keep their normal auto-setup logic while receiving
a macOS-compatible video source without driver code changes.

For camera twins, Edge Core can also provide default bridge candidates on macOS
even when metadata has no explicit `--device` params (default driver config),
so Linux-oriented camera drivers remain compatible with minimal metadata.

To inject environment variables into a driver container, list `-e` flags inside `params`: each `-e` must be a separate element followed by its `KEY=value` string. Example:

```json
{
  "drivers": {
    "default": {
      "docker_image": "cyberwaveos/go2-native-driver",
      "params": ["-e", "MY_VAR=value", "-e", "ANOTHER_VAR=value2"]
    }
  }
}
```

Each `-e` must be its own element in the array, followed by the `KEY=value` string as the next element. This is equivalent to passing `-e MY_VAR=value` on the `docker run` command line.

This is useful for driver-specific configuration that varies per device, such as IP addresses, credentials, or feature flags that cannot be stored in the twin's `edge_configs` metadata.

### Runtime configuration for drivers (`metadata["edge_configs"]`)

Drivers and edge services should treat `metadata["edge_configs"]` as the source of truth for per-device runtime configuration.
Edge identity should be stored at `metadata["edge_fingerprint"]` (not duplicated inside `edge_configs`).

> **Runtime access**: The core passes the full twin JSON (including `metadata`) to every driver via the `CYBERWAVE_TWIN_JSON_FILE` environment variable. Drivers can read `edge_configs` from that file at startup to obtain per-device settings — for example, selecting the right camera source or IP address for the current machine. This is the recommended way to pass device-specific configuration to a driver without hardcoding values in the image.

- Type: object/dictionary
- Value: binding object (`object`)

Canonical shape:

```json
{
  "edge_fingerprint": "macbook-pro-a1b2c3d4e5f6",
  "edge_configs": {
    "camera_config": {
      "camera_id": "front",
      "source": "rtsp://user:pass@192.168.1.20/stream",
      "fps": 10,
      "resolution": "VGA",
      "camera_type": "cv2"
    }
  }
}
```

Field notes:

- `edge_fingerprint`: fingerprint of the edge serving this twin (recommended).
- `camera_config`: per-device camera/runtime config consumed by drivers.

Avoid storing transient runtime state such as `edge_uuid`, `registered_at`, `last_sync`, `last_ip_address`, or `status_data` inside `edge_configs`.

Backward compatibility:

- Older records may use a legacy map shape (`edge_configs[fingerprint] = {...}`).
- Older records may store camera settings in `cameras[0]` or as top-level fields.
- New writers should prefer `camera_config` under `edge_configs`.
- Do not rely on `PUT /api/v1/edges/{uuid}/twins/{twin_uuid}/camera-config`; it is deprecated. Update twin metadata instead.

## Advanced usage

### Manual install and troubleshooting

```bash
# Install the Buildkite package signing key
curl -fsSL "https://packages.buildkite.com/cyberwave/cyberwave-edge-core/gpgkey" | gpg --dearmor -o /etc/apt/keyrings/cyberwave_cyberwave-edge-core-archive-keyring.gpg

# Configure the Apt source
echo -e "deb [signed-by=/etc/apt/keyrings/cyberwave_cyberwave-edge-core-archive-keyring.gpg] https://packages.buildkite.com/cyberwave/cyberwave-edge-core/any/ any main\ndeb-src [signed-by=/etc/apt/keyrings/cyberwave_cyberwave-edge-core-archive-keyring.gpg] https://packages.buildkite.com/cyberwave/cyberwave-edge-core/any/ any main" \
  > /etc/apt/sources.list.d/buildkite-cyberwave-cyberwave-edge-core.list

# Run Edge Core (performs startup checks and starts drivers + worker container)
cyberwave-edge-core

# Show status, credentials and MQTT connectivity (read-only)
cyberwave-edge-core status

# Show version
cyberwave-edge-core --version

# Worker container management (also available via `cyberwave worker …`)
cyberwave-edge-core worker start      # Start the worker container
cyberwave-edge-core worker stop       # Stop the worker container
cyberwave-edge-core worker restart    # Restart the worker container
cyberwave-edge-core worker status     # Show container state, workers, cached models, and health
cyberwave-edge-core worker health     # Show detailed restart history and circuit-breaker state
cyberwave-edge-core worker logs       # Stream worker container logs
cyberwave-edge-core worker logs --no-follow  # Print recent logs without following
```

Preview builds from `dev` / `staging` CI are published as **separate Debian packages** in the same apt repo: `cyberwave-edge-core-dev` and `cyberwave-edge-core-staging`. `apt install cyberwave-edge-core` only pulls tagged releases; use one of the channel packages explicitly when you want those binaries (the packages conflict because they ship the same `/usr/bin/cyberwave-edge-core`).

On non-apt platforms, prerelease Python wheels are published to the Buildkite Python registry and consumed automatically by `cyberwave edge install --channel dev|staging`. Stable pip installs continue to use the public PyPI release.

### Environment variables

Run against a different environment/base URL:

```bash
export CYBERWAVE_ENVIRONMENT="yourenv"
export CYBERWAVE_BASE_URL="https://yourbaseurl"
cyberwave-edge-core
```

Control log verbosity (default: `INFO`). Edge-core forwards any
`CYBERWAVE_*` env var it sees in its own process environment into each
worker container at `docker run` time, so the same knobs work whether
they're set in the systemd unit, the CLI install env, or the shell:

```bash
# Edge-core + edge drivers (camera/UGV/etc.)
export CYBERWAVE_EDGE_LOG_LEVEL="DEBUG"
# Worker containers only — wins over CYBERWAVE_EDGE_LOG_LEVEL when set,
# so you can keep workers at INFO while a driver is at DEBUG (or vice versa).
export CYBERWAVE_WORKER_LOG_LEVEL="INFO"
cyberwave-edge-core
```

Or pass env vars to the CLI installer:

```bash
sudo CYBERWAVE_ENVIRONMENT="yourenv" CYBERWAVE_BASE_URL="https://yourbaseurl" CYBERWAVE_MQTT_HOST="yourmqtt" cyberwave edge install
```

### Disk usage: container logs and cleanup

Docker's default `json-file` log driver has no size limit, and every container
Edge Core launches runs under `--restart unless-stopped`. On a device with a
small disk — a Raspberry Pi on a 16 GB uSD, say — a chatty driver can fill the
card in days. Edge Core therefore caps each container's log at launch rather
than relying on the host's `/etc/docker/daemon.json`.

| Variable | Default | Purpose |
| --- | --- | --- |
| `CYBERWAVE_DOCKER_LOG_MAX_SIZE` | `10m` | Max size per container log file before rotation. Set to `off` to emit no log flags at all and inherit the host daemon's configuration — use this if you ship logs through a daemon-wide driver such as `journald` or `fluentd`. |
| `CYBERWAVE_DOCKER_LOG_MAX_FILE` | `3` | Rotated files kept per container (so 30 MB per container by default). |
| `CYBERWAVE_DRIVER_LOG_COLD_START_TAIL` | `500` | Backlog lines replayed when Edge Core starts following an already-running container's logs. `0` restores the previous unbounded replay. |

Per-container flags override `daemon.json`, so the defaults match what the Pi
image already sets daemon-wide — installing Edge Core never raises a cap the
host had already chosen. Raise them explicitly if you want more history.

A driver's catalog `docker_run_params` win, at the granularity they were set:
an explicit `--log-driver` means Edge Core adds nothing at all, while an
explicit `--log-opt max-size` (with no driver) keeps that value and still gets
the default `max-file` bound.

Periodic cleanup prunes stopped Cyberwave containers (every 30 min) plus unused
images and anonymous volumes (every 3 h):

| Variable | Default | Purpose |
| --- | --- | --- |
| `CYBERWAVE_SKIP_PERIODIC_DOCKER_CLEANUP` | unset | Set to `1` to disable all periodic pruning. |
| `CYBERWAVE_SD_CARD_CLEANUP_USAGE_PERCENT` | `80` | When Docker's data root is on an SD card, routine pruning is deferred to limit flash wear — but runs anyway once that filesystem reaches this percentage full. |
| `CYBERWAVE_CONTAINER_PRUNE_INTERVAL_SECONDS` | `1800` | Stopped-container prune interval. |
| `CYBERWAVE_IMAGE_PRUNE_INTERVAL_SECONDS` | `10800` | Unused-image prune interval. |
| `CYBERWAVE_VOLUME_PRUNE_INTERVAL_SECONDS` | `10800` | Anonymous-volume prune interval. Named volumes are never pruned. |

Volume pruning requires Docker 23.0 or newer, where `docker volume prune`
removes only anonymous volumes. On older daemons the same command would also
delete unused *named* volumes belonging to other workloads on the host, so Edge
Core skips it and prunes containers and images only. The `docker-ce` packages
our installer and Pi image use are well past 23.0; the distro `docker.io`
package is not.

If a device is running out of space, start here:

```bash
docker system df -v
sudo ls -lSh /var/lib/docker/containers/*/*-json.log | head
du -sh ~/.cyberwave/models/
journalctl --disk-usage
```

## Local development (from this folder)

You can develop both the **Cyberwave CLI** and **Edge Core** from the `cyberwave-edge-core` directory using a single virtual environment that has the monorepo SDK, CLI, and edge-core installed in editable mode.

### One-time setup

From `cyberwave-edge-core/`:

```bash
# Create and activate a venv (e.g. .venv in this folder)
python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

# Install SDK, CLI, and Edge Core in editable mode (order matters: SDK first)
pip install -e ../cyberwave-sdks/cyberwave-python
pip install -e ../cyberwave-clis/cyberwave-python-cli/"[build]"
pip install -e ".[build]"
```

**Generate the SDK REST client (required for editable SDK).** The SDK’s `cyberwave.rest` package is generated from the backend OpenAPI spec and is not committed. If you see `ImportError: cannot import name 'DefaultApi' from 'cyberwave.rest'`:

1. Start the backend: `cd ../cyberwave-backend && docker compose -f local.yml up -d` (wait until healthy).
2. From the repo root, generate the REST client:
   ```bash
   cd cyberwave-sdks && ./python-sdk-gen.sh sdk --host localhost:8000
   ```
3. Re-run the `pip install -e` steps above if you already installed; the editable SDK will then include the generated `cyberwave/rest` code.

### Run CLI and Edge Core

After activating the venv, both commands are on your PATH:

```bash
# CLI
cyberwave --help
cyberwave login --email boss@cyberwave.com --password iamnottheboss
cyberwave edge install --help

# Edge Core
cyberwave-edge-core --help
cyberwave-edge-core status
cyberwave-edge-core
```

**Target backend:** If you do not set `CYBERWAVE_BASE_URL`, the CLI and Edge Core use the default **production** API (`https://api.cyberwave.com`). To use your local backend instead:

```bash
export CYBERWAVE_BASE_URL=http://localhost:8000
export CYBERWAVE_MQTT_HOST=localhost
export CYBERWAVE_ENVIRONMENT=local
```

### Paths from this folder

| What       | Path (from `cyberwave-edge-core/`)         |
| ---------- | ------------------------------------------ |
| Repo root  | `..`                                       |
| Python SDK | `../cyberwave-sdks/cyberwave-python`       |
| CLI        | `../cyberwave-clis/cyberwave-python-cli`   |

Edit code in any of those directories; the editable installs pick up changes (no reinstall needed for Python changes).

## Testing

CI (`.github/workflows/edge-core-test-and-push.yml`) runs the whole unit suite, `pytest tests/ --ignore=tests/integration`, and then the Docker integration tests separately. Run the same thing locally from this folder:

```bash
pytest tests/ --ignore=tests/integration -q
```

The systemd behaviour (readiness, the watchdog, bounded start and stop, upgrades of the Debian package) can only be checked under a real systemd. `scripts/test_deb_upgrade_under_systemd.sh` boots one in a container, installs the package given by `PACKAGE_FILE` and checks install, upgrade, hold and update. It then runs the lifecycle test files of the CLI and of Edge Core inside that systemd, with `CYBERWAVE_TEST_SYSTEMD_SYSTEM=1` and `CYBERWAVE_REQUIRE_SYSTEMD_TESTS=1` so that tests needing systemd fail instead of skipping. The Edge Core checkout is mounted read-only. `LIFECYCLE_ONLY=1` skips the Debian checks and needs no package, which is the quick way to check a lifecycle change:

```bash
LIFECYCLE_ONLY=1 bash scripts/test_deb_upgrade_under_systemd.sh
PACKAGE_FILE=path/to/package.deb bash scripts/test_deb_upgrade_under_systemd.sh
```

## Contributing

Contributions are welcome. Please open an issue to discuss bugs or feature requests, and submit a pull request when you are ready.

## Community and Documentation

- Documentation: https://docs.cyberwave.com
- Community (Discord): https://discord.gg/dfGhNrawyF
- Issues: https://github.com/cyberwave-os/cyberwave-edge-core/issues
