# Testing

This project separates copyable examples from verification configs:

- **Examples** teach one idea each. They live in
  `src/rosotacom/resources/examples/` and are copied with
  `rosotacom examples create`.
- **Test-configs** cover combinations. The heartbeat RMW matrix lives in
  `tests/sessions/rmw_matrix` and is generated by
  `tests/sessions/generate_rmw_matrix.py`.

## Test Layers

| Layer | Location | Docker | Network | Runs in |
|---|---|---|---|---|
| host / unit | `tests/unit` | no | none | draft + ready PR |
| contract | `tests/contract` | no | none | ready PR |
| local smoke | `tests/e2e` | yes | one host, isolated bridge | merge gate |
| performance regression gate | benched set (`benchmark row`) | yes | one host, emulated profiles | merge gate (one row) + blocking nightly matrix |
| full local suite | examples + `tests/sessions` | yes | one host, isolated bridge | nightly + manual |
| external OTA | operator deployment | yes | multiple machines | manual promotion gate |

Ready PRs run the non-Docker checks, package validation, and the local smoke
slice. The full local suite also runs nightly, as does the blocking benchmark
regression matrix ([performance-bands.md](performance-bands.md)). The external
OTA gate remains operator-started before a promotion to `main`.

## Expectations

Per-topic `expect` blocks are behavioral contracts in `session-definition.yaml`.
The status overview consumes them at runtime, and `rosotacom test` asserts the
latest `logs/<peer>/status/status.json` self-report:

```yaml
topics:
  b_to_a:
    - topic: /costmap/costmap
      expect:
        hz: { min: 1, max: 5 }
        latency_ms: { max: 500 }
        loss_pct: { max: 2, stage: com_in }
```

Heartbeat expectations live under `shared.heartbeat.expect`, because heartbeat
topics are generated by the framework rather than listed under `topics:`.
`loss_pct` is exact sequence-gap loss for echo heartbeats and wrapped
`OtaStamped` streams; wrapped streams normally measure it at inbound `com_in`.

`expect.smoke_probe: false` keeps a topic in the generated OTA contract while
excluding it from the synthetic local smoke publisher/probe set and from the
local smoke self-report assertion. Use it only for redundant replay-shape topics
when a nearby representative stream already proves that path in CI.

### Whole-Bag Expect Generation

For replay gates, a bag's `metadata.yaml` is a ground-truth source: it records
which topics exist, how many messages each contains, the bag duration, and the
offered QoS. Generate a mergeable `expect:` fragment for the topics a session
carries with:

```bash
rosotacom expect from-bag ./native_bag --session remote_assist --out whole-bag-expect.yaml
```

The generator reads only rosbag2 metadata. It does not rewrite the session in
place. The output is a YAML fragment under `topics:` with one commented
derivation per generated entry, so it can be reviewed and merged into a
session-definition manually.

Stream topics receive `min_count` plus `completeness.min_ratio` and
`completeness.vs_bag_ratio`. The count and bag-relative rate are scaled by
intended session shaping before the threshold is emitted: `drop` applies its
keep fraction, and `throttle_hz` caps the expected post-drop rate. That means
planned decimation is not counted as OTA loss. The default `--min-ratio 0.9`
leaves a safety margin for a healthy link; tune it per gate instead of
committing a brittle 100% threshold.

**`vs_bag_ratio` is replay-only, and the run refuses to pretend otherwise.** It
compares the delivered rate to the source's native rate, which only the bag
knows, so `rosotacom test` needs `--bag <dir>` to evaluate it. Until #214 a run
without one skipped those assertions silently and still printed `TEST OK` — a
session could declare completeness on every topic and have none of it checked.
`test` now exits non-zero naming the topics instead. If a session genuinely is
not replay-driven, drop the assertion rather than running without the bag: a
skipped check reads as coverage, which is the failure mode these contracts
exist to catch.

Transient-local or explicitly latched topics become `mode: latched`, because the
contract is that the held value arrives, not that the topic keeps ticking.
Sparse volatile topics become `mode: existence`; the current expect model has no
separate finite "delivered once and may now be stale" mode, so those entries
assert graph presence without changing evaluator semantics.

Generated whole-bag contracts complement hand-curated contracts. Keep the
hand-curated session for the operator-facing steady-state checks that humans
understand and maintain. Use a generated whole-bag variant when the question is
"does this replay carry every session topic from the reference recording, after
the session's intended shaping?" Regenerate the fragment when the reference bag
or the session pipeline changes.

## Local Check Eligibility

OTA suite membership is the default. There is no `multi_machine` marker.

The only per-session test axis is whether the session can be checked quickly on
one host. `rosotacom.cli.local_check_sessions()` derives that from distinct
per-peer local `domain_id` values:

```yaml
peer_settings:
  a: { domain_id: 46 }
  b: { domain_id: 47 }
```

Configs that cannot be faked locally opt out explicitly:

```yaml
local_check: false
```

That opt-out does not remove the config from the OTA suite. It only removes it
from one-host smoke.

## Local Smoke

`just test-e2e-smoke` runs the entire set of local-checkable curated examples. In CI and for local convenience, you can run specific E2E test slices or nodes:

- `just test-e2e-slice heartbeat`: the heartbeat matrix (and the opt-in RMW matrix).
- `just test-e2e-slice chatter`: the native chatter scenario, smoke, and debug rig.
- `just test-e2e-slice occupancy-grid-dds`: compressed occupancy grid over direct DDS.
- `just test-e2e-slice occupancy-grid-zenoh`: compressed occupancy grid over Zenoh.
- `just test-e2e-slice sized-payload-fastdds`: sized payload over Fast DDS.
- `just test-e2e-slice sized-payload-zenoh`: sized payload over Zenoh.
- `just test-e2e-slice remote-assist`: the full anonymized remote-assist rig.
- `just test-e2e-slice remote-assist-costmap`: its costmap single-stream cut.
- `just test-e2e-slice remote-assist-camera`: its camera single-stream cut.
- `just test-e2e-slice runtime-tools`: link latency and live timeline stepping.
- `just test-e2e-slice media`: anonymization and video quality.
- `just test-e2e-slice concurrency-parallel`: independent smoke runs coexist.
- `just test-e2e-slice concurrency-conflict`: a conflicting smoke run is rejected safely.
- `just test-e2e-slice benchmark-ab`: the A/B reliability verdict.
- `just test-e2e-slice benchmark-replay`: loss boundaries against costmap replay.
- `just test-e2e-slice benchmark-capacity`: the band-asserted capacity rows.
- `just test-e2e-slice benchmark-capacity-cases`: the capacity good/bad-case verdicts.
- `just test-e2e-node <nodeid>`: Reruns a specific pytest node ID directly.
- `just e2e-slice-costs`: prints what each slice costs and its distance to the floor.

The slices are grouped by diagnostic theme and split where transport, stream,
or behavior gives a useful first failure distinction. Measured cost decides
when a theme needs another runner, not which unrelated tests should share one.
`E2E_SLICES` in `tests/e2e/conftest.py` decides which slice owns which test;
see [ci.md](ci.md#diagnostic-e2e-slices) before changing one.

The generated RMW matrix is available as an opt-in full local slice:

```bash
ROSOTACOM_RUN_E2E=1 ROSOTACOM_RUN_FULL_E2E=1 \
  .venv/bin/python -m pytest -q tests/e2e/test_smoke.py -m e2e
```
Or run a specific matrix session:
```bash
just test-e2e-rmw <session_name>
```

## OTA Gate

Multi-machine OTA verification needs a real two-host link, so public CI does not
run it. A repository operator runs the full suite manually before promoting a
candidate to `main`.

A generic OTA run:

1. stages and installs the current rosotacom source on both hosts,
2. stages the complete active project and any external deployment file,
3. resolves named hosts or raw peer addresses,
4. starts identity `a` on host A and identity `b` on host B,
5. starts synthetic publishers with `rosotacom publish-test-topics` where needed,
6. asserts delivery with `rosotacom test`,
7. asserts local-topic isolation with `rosotacom probe-publish` and
   `rosotacom probe-check --expect absent`,
8. collects each host's `session-instances/` logs and cleans up.

Use `rosotacom ota-smoke TARGET --peer-address ... --peer-ssh ...` for a
configuration-free operator run, or map peers to named hosts with `--peer`
through the active project's deployment file. Add `--interactive` to open a
local control tmux for the same run; each peer gets an attachable
communication/catmux window with a live status pane below it, and scenario
targets also get native application-container windows. Stop it with
`rosotacom ota-smoke TARGET --interactive --stop`.

External runners should use `ota_suite_sessions()` to discover the scenario set
from examples and generated test-configs instead of hardcoding transport names.

## Idealized Camera Load (GOP + Jitter)

To study packet burst and jitter effects in isolation without relying on heavy bag replays, you can configure a synthetic "idealized camera" load using the `--size-pattern` and publish-interval jitter parameters.

### Translating Measured Stream Stats to Parameters

Suppose you analyze a real camera stream (similar to the processed handoff trace in [docs/ffmpeg-keyframes.md](ffmpeg-keyframes.md)) and measure the following:
* **Frame Rate**: Nominal rate is 10 Hz (nominal period of 100 ms).
* **GOP (Group of Pictures)**: GOP size is 5.
* **Frame Sizes**: Keyframe (I-frame) average is ~43 KiB. The 4 delta frames (P-frames) average is ~4 KiB.
* **Interval Jitter**: Standard deviation of frame intervals is measured at ~20 ms.

You can translate these metrics directly into the following CLI arguments for the `benchmark` tool:

1. **Nominal Rate**: `--rate-hz 10.0`
2. **GOP-Shaped Sizes**: `--size-pattern 1x43KiB+4x4KiB` (which repeats cyclically: `43 KiB, 4 KiB, 4 KiB, 4 KiB, 4 KiB`).
3. **Interval Jitter**: `--interval-jitter-ms 20.0 --interval-jitter-seed 42` (specifies a 20 ms standard deviation Gaussian noise around the 100 ms nominal period, seeded with a deterministic seed of 42 to ensure the publish schedule is identical across runs).

This maps the measured network footprint of the camera stream to a clean, reproducible synthetic load.
