Metadata-Version: 2.3
Name: gauntlet-robotics
Version: 0.2.0
Summary: An evaluation harness for learned robot policies.
Keywords: robotics,evaluation,policy,mujoco,regression-testing
Author: Gauntlet contributors
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: mujoco>=3.2,<5
Requires-Dist: gymnasium>=1.0,<2
Requires-Dist: pydantic>=2.7,<3
Requires-Dist: typer>=0.12,<1
Requires-Dist: numpy>=1.26,<3
Requires-Dist: pandas>=2.2,<4
Requires-Dist: jinja2>=3.1,<4
Requires-Dist: pyyaml>=6.0,<7
Requires-Dist: rich>=13.7,<17
Requires-Dist: transformers>=4.40,<5 ; extra == 'dt'
Requires-Dist: torch>=2.2,<3 ; extra == 'dt'
Requires-Dist: huggingface-hub>=0.20,<1 ; extra == 'dt'
Requires-Dist: genesis-world>=0.4,<0.5 ; extra == 'genesis'
Requires-Dist: torch>=2.2,<3 ; extra == 'genesis'
Requires-Dist: lerobot>=0.4,<1 ; extra == 'groot'
Requires-Dist: transformers>=4.40,<5 ; extra == 'groot'
Requires-Dist: pillow>=10.0,<12 ; extra == 'groot'
Requires-Dist: torch>=2.2,<3 ; extra == 'hf'
Requires-Dist: transformers>=4.40,<5 ; extra == 'hf'
Requires-Dist: timm>=0.9.10,<2 ; extra == 'hf'
Requires-Dist: tokenizers>=0.19,<1 ; extra == 'hf'
Requires-Dist: pillow>=10.0,<12 ; extra == 'hf'
Requires-Dist: isaacsim>=5.0,<6 ; extra == 'isaac'
Requires-Dist: lerobot[smolvla]>=0.4,<1 ; extra == 'lerobot'
Requires-Dist: transformers>=4.40,<5 ; extra == 'lerobot'
Requires-Dist: pillow>=10.0,<12 ; extra == 'lerobot'
Requires-Dist: mlflow>=2.10,<4 ; extra == 'mlflow'
Requires-Dist: torch>=2.2,<3 ; extra == 'monitor'
Requires-Dist: pillow>=10.0,<12 ; extra == 'monitor'
Requires-Dist: pyarrow>=15.0,<22 ; extra == 'parquet'
Requires-Dist: lerobot[pi]>=0.4,<1 ; extra == 'pi0'
Requires-Dist: transformers>=4.40,<5 ; extra == 'pi0'
Requires-Dist: pillow>=10.0,<12 ; extra == 'pi0'
Requires-Dist: pybullet>=3.2,<4 ; extra == 'pybullet'
Requires-Dist: torch>=2.2,<3 ; extra == 'rdt'
Requires-Dist: transformers>=4.40,<5 ; extra == 'rdt'
Requires-Dist: huggingface-hub>=0.20,<1 ; extra == 'rdt'
Requires-Dist: pillow>=10.0,<12 ; extra == 'rdt'
Requires-Dist: torch>=2.0,<3 ; extra == 'realsim-gsplat'
Requires-Dist: gsplat>=1.0,<2 ; extra == 'realsim-gsplat'
Requires-Dist: dtw-python>=1.4,<2 ; extra == 'trajectory-taxonomy'
Requires-Dist: imageio[ffmpeg]>=2.34,<3 ; extra == 'video'
Requires-Dist: wandb>=0.16,<1 ; extra == 'wandb'
Requires-Python: >=3.11
Project-URL: Changelog, https://github.com/mhussainahmad/gauntlet/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/mhussainahmad/gauntlet/blob/main/README.md
Project-URL: Homepage, https://github.com/mhussainahmad/gauntlet
Project-URL: Issues, https://github.com/mhussainahmad/gauntlet/issues
Project-URL: Repository, https://github.com/mhussainahmad/gauntlet
Provides-Extra: dt
Provides-Extra: genesis
Provides-Extra: groot
Provides-Extra: hf
Provides-Extra: isaac
Provides-Extra: lerobot
Provides-Extra: mlflow
Provides-Extra: monitor
Provides-Extra: parquet
Provides-Extra: pi0
Provides-Extra: pybullet
Provides-Extra: rdt
Provides-Extra: realsim-gsplat
Provides-Extra: ros2
Provides-Extra: trajectory-taxonomy
Provides-Extra: video
Provides-Extra: wandb
Description-Content-Type: text/markdown

# Gauntlet

> An evaluation harness for learned robot policies.

Gauntlet answers a single question for VLA / diffusion / scripted policies:

> *"How does this policy fail, and has the latest checkpoint regressed against the last one?"*

It wraps any policy behind a uniform adapter, runs it across a parameterized
suite of MuJoCo perturbations (lighting, camera pose, textures, clutter,
initial conditions), and produces a structured report that **breaks failures
down by axis** instead of hiding them in an aggregate mean.

See [`GAUNTLET_SPEC.md`](./GAUNTLET_SPEC.md) for the full design.

---

## See a real report before installing

A public reference benchmark lives at
**<https://mhussainahmad.github.io/gauntlet/>** — two policies on the
bundled smoke suite, regenerated on every release, with the
`gauntlet compare` and `gauntlet diff` deltas surfaced. Open the
baseline / regressed `report.html` to see what the failure-cluster-first
layout actually looks like.

To regenerate it locally:

```bash
python scripts/generate_reference_benchmark.py --out ./benchmarks/local/
open ./benchmarks/local/index.html
```

## Install

```bash
pip install gauntlet-robotics           # core (MuJoCo only, torch-free)
pip install 'gauntlet-robotics[hf]'        # + OpenVLA / HuggingFace adapter
pip install 'gauntlet-robotics[lerobot]'   # + SmolVLA / π0 / diffusion adapters
pip install 'gauntlet-robotics[pybullet]'  # + PyBullet backend
pip install 'gauntlet-robotics[genesis]'   # + Genesis backend
pip install 'gauntlet-robotics[isaac]'     # + Isaac Sim backend (CUDA required)
pip install 'gauntlet-robotics[monitor]'   # + runtime drift detector (torch)
pip install 'gauntlet-robotics[ros2]'      # + ROS 2 publish / record (rclpy via system pkg)
```

`uv` users: `uv add gauntlet-robotics` (same extras). Stand-alone CLI:
`uv tool install gauntlet-robotics` then `gauntlet --help`. Requires Python
≥3.11.

## Status

Phase 1 (MVP) and Phase 2 (real-policy adapters + runtime
observability) are shipped. Phase 1 covers the tabletop env, the
seven perturbation axes, the parallel Runner, the breakdown-first
HTML report, and the core `gauntlet run / report / compare` CLI.
Phase 2 adds the PyBullet / Genesis / Isaac Sim backends, OpenVLA
and SmolVLA adapters, runtime drift detection (`monitor`), ROS 2
publishing + recording, multi-camera observations, structured
per-axis report diffs (`gauntlet diff`), incremental rollout caching,
and the entry-point-based plugin system for third-party policies
and envs.

Phase 3 (fleet-scale tooling) is **partially shipped**: the
fleet-wide failure-mode aggregator (`gauntlet aggregate`), the
self-contained web dashboard, and the real-to-sim scene-ingestion
input pipeline are all live. The real-to-sim *renderer* itself is
deferred — `RealSimRenderer` lands as a `typing.Protocol` so a
gaussian-splatting (or other) renderer plugin can slot in without
touching the schema.

`0.2.0` is the first PyPI release: `pip install gauntlet-robotics`. From this
release onward, the documented public surface follows
[Semantic Versioning](https://semver.org/spec/v2.0.0.html) — the
full contract (which symbols are public, the on-disk schemas, the
CLI flags, and the deprecation policy) is in
[`docs/stability.md`](./docs/stability.md). Pin
`gauntlet>=0.2,<0.3` in your `pyproject.toml` and CI will not break
on a patch release.

## Backends

Gauntlet ships four simulator backends. The Suite YAML's `env:` key
is the dispatch: `tabletop` uses MuJoCo (default, ships in the core
install); `tabletop-pybullet`, `tabletop-genesis`, and
`tabletop-isaac` each live behind an optional extra.

| `env:` slug          | Simulator | Install                                  | Observations |
|----------------------|-----------|------------------------------------------|--------------|
| `tabletop`           | MuJoCo    | `uv sync` (core)                         | State + render-on-demand |
| `tabletop-pybullet`  | PyBullet  | `uv sync --extra pybullet`               | State + render-on-demand |
| `tabletop-genesis`   | Genesis   | `uv sync --extra genesis`                | State + render-on-demand |
| `tabletop-isaac`     | Isaac Sim | `uv sync --extra isaac` (GPU required)   | State-only (rendering follow-up) |

The four backends share action/observation spaces byte-for-byte and
the canonical 7 perturbation axes. They are **not** numerically
identical: same policy + same seed on `tabletop` vs `tabletop-pybullet`
vs `tabletop-genesis` vs `tabletop-isaac` produces semantically similar
but numerically different trajectories. Running `gauntlet compare`
across backends measures simulator drift, not policy regression; the
CLI requires `--allow-cross-backend` to proceed.

The `tabletop-isaac` backend wraps NVIDIA Omniverse Kit and **requires
a CUDA-capable RTX-class GPU at runtime**. The `[isaac]` extra resolves
on CPU-only machines but the Kit bootstrap inside `IsaacSimTabletopEnv.__init__`
fails without a GPU. CI tests use a `sys.modules`-injected fake
`isaacsim` namespace and do NOT install this extra; live execution
needs a developer GPU workstation. The state-only first cut declares
the four cosmetic axes (`lighting_intensity`, `camera_offset_x`,
`camera_offset_y`, `object_texture`) `VISUAL_ONLY_AXES` so cosmetic-only
sweeps are rejected at suite-load time on this backend until the
rendering follow-up RFC lands.

Image observations are available on all three backends via
`render_in_obs=True` / `render_size=(H, W)` on the env constructor
(`TabletopEnv`, `PyBulletTabletopEnv`, or `GenesisTabletopEnv`).
PyBullet uses a headless, deterministic TINY rasteriser; Genesis uses
its default CPU Rasterizer (pyrender-backed). The emitted `obs["image"]`
Box has shape / dtype / bounds byte-identical across backends, so VLA
adapters (OpenVLA, SmolVLA) work on any of them by swapping only the
env factory. Pixel values explicitly differ (different rasterisers —
semantic parity only). All seven perturbation axes produce observable
deltas on the rendered image on every backend; `VISUAL_ONLY_AXES` is
empty everywhere.

For multi-view policies (SmolVLA, ACT, Diffusion Policy — anything
that consumes paired wrist + side + overhead frames), pass
`cameras=[CameraSpec(...), ...]` to `TabletopEnv` or
`PyBulletTabletopEnv` instead. Each spec lands in
`obs["images"][name]`; `obs["image"]` stays populated as an alias to
the first camera so single-view consumers (the runner's video
recorder, OpenVLA-style adapters) keep working unchanged. The
single-camera default (`cameras=None`) is byte-identical to the
phase-1 contract — see
[`docs/polish-exploration-multi-camera.md`](./docs/polish-exploration-multi-camera.md)
for the full design and
[`examples/evaluate_multi_camera.py`](./examples/evaluate_multi_camera.py)
for a worked example.

See [`docs/phase2-rfc-005-pybullet-adapter.md`](./docs/phase2-rfc-005-pybullet-adapter.md)
for the full PyBullet backend design,
[`docs/phase2-rfc-006-pybullet-rendering.md`](./docs/phase2-rfc-006-pybullet-rendering.md)
for PyBullet's image-observation follow-up,
[`docs/phase2-rfc-007-genesis-adapter.md`](./docs/phase2-rfc-007-genesis-adapter.md)
for the Genesis backend design,
[`docs/phase2-rfc-008-genesis-rendering.md`](./docs/phase2-rfc-008-genesis-rendering.md)
for the Genesis image-observation follow-up, and
[`docs/phase2-rfc-009-isaac-sim-adapter.md`](./docs/phase2-rfc-009-isaac-sim-adapter.md)
for the Isaac Sim backend design.

## Quickstart

Three commands reproduce the end-to-end example against the bundled
smoke suite (3 lighting intensities x 2 cube textures x 4 episodes = 24
rollouts; finishes in seconds on a laptop):

```bash
uv sync
uv run gauntlet run examples/suites/tabletop-smoke.yaml --policy random --out out/
open out/report.html  # macOS: open ; Linux: xdg-open ; Windows: start
```

Artefacts land in `out/`: `episodes.json` (one record per rollout),
`report.json` (analysed breakdowns), and `report.html` — a self-contained
report leading with the failure-clusters table, then per-axis bar charts,
then 2D heatmaps of axis combinations. The smoke suite is intentionally
tiny; for the canonical 4-axis x 144-cell x 1440-rollout shape, swap the
YAML path for `examples/suites/tabletop-basic-v1.yaml`. See
[`GAUNTLET_SPEC.md`](./GAUNTLET_SPEC.md) for the full design and
[`examples/evaluate_random_policy.py`](./examples/evaluate_random_policy.py)
for the equivalent invocation via the public Python API. For the
Genesis backend, `uv sync --extra genesis` then
[`examples/evaluate_random_policy_genesis.py`](./examples/evaluate_random_policy_genesis.py)
drives the same smoke suite against `tabletop-genesis`.

The Suite YAML's `sampling:` key picks the perturbation grid strategy:
the default `cartesian` enumerates the full Cartesian product of the
declared axes (the historical behaviour, byte-identical to every
existing suite), while `latin_hypercube` and `sobol` each draw
`n_samples` points without enumerating the full grid. For five axes at
five steps, cartesian = 3,125 cells; LHS or Sobol at `n_samples: 32`
covers the same hypercube at ~98x fewer rollouts. The two quasi-random
samplers trade off differently:

- `latin_hypercube` (McKay 1979) gives **perfect per-axis marginal
  stratification** — every axis covers exactly `n_samples` distinct
  strata. Joint coverage across axis pairs is essentially random.
- `sobol` (Joe-Kuo 6.21201 direction numbers, `skip=1`) gives
  **low-discrepancy joint coverage** — Sobol projections onto any
  axis pair are also quasi-uniform, at the cost of slightly worse
  per-axis marginal histograms than LHS.

Use Sobol when the failure mode you suspect is a 2-axis (or higher)
interaction; use LHS when single-axis sweeps are what you need to
cover. See
[`examples/suites/tabletop-lhs-smoke.yaml`](./examples/suites/tabletop-lhs-smoke.yaml)
and [`examples/evaluate_random_policy_lhs.py`](./examples/evaluate_random_policy_lhs.py)
for an LHS end-to-end demo, and
[`docs/polish-exploration-sobol-sampler.md`](./docs/polish-exploration-sobol-sampler.md)
for the Sobol design note (discrepancy targets, direction-number
table, skip rationale).

Once you have multiple runs (different seeds, policy revisions, or
backends), `gauntlet aggregate <runs-dir> --out fleet/` rolls every
`report.json` recursively under `<runs-dir>` into a single fleet
meta-report — `fleet/fleet_report.json` plus a self-contained
`fleet/fleet_report.html` leading with the persistent failure
clusters that survive across runs (clusters appearing in at least
`--persistence-threshold` of the runs, default `0.5`). See
[`examples/aggregate_runs.py`](./examples/aggregate_runs.py) for the
equivalent invocation via the Python API and
[`docs/phase3-rfc-019-fleet-aggregate.md`](./docs/phase3-rfc-019-fleet-aggregate.md)
for the algorithm.

### Using a real VLA

- Install the HF extras: `uv sync --extra hf` (pulls torch / transformers / pillow; core installs stay torch-free).
- See [`examples/evaluate_openvla.py`](./examples/evaluate_openvla.py) for the ≤20-line OpenVLA-7B factory.
- Image-conditioned policies need a rendered frame — construct `TabletopEnv(render_in_obs=True)` so `obs["image"]` is emitted.
- **SmolVLA — read the warning first.** `lerobot/smolvla_base` is
  pretrained on the SO-100 / SO-101 follower arm with **6-D
  joint-position** actions; TabletopEnv is a **7-D EE-twist + gripper**
  env. Zero-shot success on the smoke suite is **~0% by embodiment
  mismatch — this is NOT a Gauntlet bug.** The example exists so users
  who already have a TabletopEnv-compatible fine-tune know how to wire
  the adapter (≤20 lines). For a first run that demonstrates the
  harness end-to-end on a policy that solves the env, use
  [the reference benchmark](#see-a-real-report-before-installing)
  instead.
- With a fine-tune in hand: `uv sync --extra lerobot`;
  see [`examples/evaluate_smolvla.py`](./examples/evaluate_smolvla.py)
  (pass `--action-remap` / override `camera_keys` if your fine-tune
  changed them). For the PyBullet backend,
  `uv sync --extra lerobot --extra pybullet` and run
  [`examples/evaluate_smolvla_pybullet.py`](./examples/evaluate_smolvla_pybullet.py).
  Set `GAUNTLET_SUPPRESS_SMOLVLA_WARNING=1` to silence the runtime
  banner once you've confirmed the embodiment fits.

### Runtime drift detection

Optional Phase 2 add-on (`[monitor]` extra). Given a reference sweep of a
known-good policy, fit a small observation autoencoder and score a
candidate sweep's trajectories against it — per-episode reconstruction
error + per-dim action-std surface OOD rollouts. `gauntlet run
--record-trajectories <dir>` dumps per-episode NPZ sidecars;
`gauntlet monitor train <dir> --out <ae_dir>` fits the AE; `gauntlet
monitor score <episodes.json> <dir> --ae <ae_dir> --out drift.json`
writes the sidecar. The three-step workflow is scripted end-to-end in
[`examples/evaluate_with_drift.py`](./examples/evaluate_with_drift.py);
`drift.json` is optional and orthogonal to `report.json`.

### ROS 2 integration

Optional Phase 2 add-on (`[ros2]` extra). Two halves wire gauntlet into a
ROS 2 graph:

- `gauntlet ros2 publish episodes.json --topic /gauntlet/episodes`
  serialises each Episode as JSON inside `std_msgs/msg/String` and
  publishes one message per Episode. Useful for fleet-wide failure-mode
  aggregation across many real robots running gauntlet evaluations.
- `gauntlet ros2 record --topic /robot/joint_states --out trajectory.jsonl
  --duration 30` subscribes to a real robot's topic and dumps each
  received message to a JSONL file on disk. Useful for the "real robots
  with logging" half of `GAUNTLET_SPEC.md` §7.

Because `rclpy` is **not** distributed via PyPI in its official form, the
`[ros2]` extra is empty — `uv sync --extra ros2` is a no-op beyond the
dev tooling. Install ROS 2 (Humble or Jazzy) via your system package
manager, e.g. `sudo apt install ros-humble-rclpy`, or run inside the
official Docker image (`docker run -it osrf/ros:humble-desktop`), then
source the relevant `setup.bash` before invoking `gauntlet ros2`. The
`--dry-run` flag on `gauntlet ros2 publish` short-circuits the rclpy
import so you can preview the JSON payloads without installing ROS 2.

The publisher / recorder API is documented in
[`docs/phase2-rfc-010-ros2-integration.md`](./docs/phase2-rfc-010-ros2-integration.md);
see [`examples/publish_episodes_to_ros2.py`](./examples/publish_episodes_to_ros2.py)
for the equivalent invocation via the public Python API.

### Diffing two runs

`gauntlet compare a.json b.json` answers a binary question (did `b`
regress against `a` beyond a threshold?). When you're iterating on a
checkpoint and want a structured, `git diff`-style breakdown of *what*
moved — per-axis-value rate deltas, per-cell success-rate flips, and the
failure-cluster set difference — reach for `gauntlet diff`:

```bash
uv run gauntlet diff out_a/report.json out_b/report.json
# Or feed episodes.json directly (auto-detected, parity with `compare`):
uv run gauntlet diff out_a/episodes.json out_b/episodes.json --json | jq
```

Threshold flags `--cell-flip-threshold` (default `0.10`) and
`--cluster-intensify-threshold` (default `0.5`) gate the per-cell and
per-cluster surfacings. Default output is human-readable text on stdout;
`--json` emits the full `ReportDiff` payload for downstream consumption.
See [`examples/diff_two_runs.py`](./examples/diff_two_runs.py) for the
equivalent invocation via the public Python API.

### Debugging failures with replay

Once a run has flagged an episode as failing, `gauntlet replay` re-
simulates exactly that rollout with the same seed, optionally nudging
one axis off the original grid:

```bash
uv run gauntlet replay out/episodes.json \
  --suite examples/suites/tabletop-smoke.yaml \
  --policy scripted \
  --episode-id 3:1 \
  --override lighting_intensity=1.2 \
  --out out/replay.json
```

Zero-override replay is bit-identical to the original episode; any
deviation points at a real reproducibility bug. See
[`examples/replay_failure.py`](./examples/replay_failure.py) for the
equivalent library call.

### Recording rollout videos

Failure analytics are far more actionable when a human can *watch*
the broken rollout. Opt in to the `[video]` extra to dump one MP4 per
episode and surface inline `<video>` thumbnails in the failure-
clusters table of the HTML report:

```bash
uv sync --extra video
uv run python examples/evaluate_random_policy_with_video.py --out out
# Open out/report.html — the failure-clusters table now embeds
# clickable thumbnails of every failed rollout.
```

The `[video]` extra pulls `imageio[ffmpeg]`, which bundles a static
ffmpeg binary — no system ffmpeg install required. Pass
`--only-failures` to suppress MP4 writes for successful episodes
(saves disk on long sweeps). The Runner asserts the env was
constructed with `render_in_obs=True` when `record_video=True`; the
example wires that automatically.

### Fleet dashboard

Once you've accumulated more than a few `report.json` files
(different policies, different seeds, nightly runs), eyeballing each
HTML report individually stops scaling. `gauntlet dashboard build`
materialises a self-contained static SPA that indexes every
`report.json` under a directory:

```bash
gauntlet dashboard build runs/ --out dashboard-out/
# Open dashboard-out/index.html via file:// — no web server needed.
```

The Python API is also exposed for notebook / custom-pipeline use:

```python
from pathlib import Path
from gauntlet.dashboard import build_dashboard

build_dashboard(Path("runs/"), Path("dashboard-out/"))
```

The output directory contains exactly three files (`index.html`,
`dashboard.js`, `dashboard.css`); all run data is embedded as an
inline JSON literal so the SPA opens straight off the filesystem
without tripping CORS. The dashboard surfaces an index card
(n_runs / n_episodes / mean ± std success rate), a per-run table
filterable by env / suite / policy, a time-series chart of success
rate keyed off `report.json` mtime, and per-axis aggregate bars
pooled across the matching runs. Sibling `report.html` files (from
the originating `gauntlet run`) are auto-linked from each row. See
[`docs/phase3-rfc-020-web-dashboard.md`](./docs/phase3-rfc-020-web-dashboard.md)
for the full design.

### Real-to-sim scene ingestion

The endgame for `GAUNTLET_SPEC.md` §7 is gaussian-splatting
reconstruction of customer scenes from real-robot camera dumps
straight into a renderable eval backend. Shipping the renderer
itself needs `torch` + CUDA + a multi-gigabyte training pipeline,
which violates spec §6 — so this release lands the *input pipeline*
and the *renderer extension point* only. A plugin (or a future
in-tree RFC) implements an actual renderer against the
`RealSimRenderer` Protocol without touching the schema or the CLI:

```bash
uv run gauntlet realsim ingest <frames-dir> \
  --calib <calib.json> \
  --out <scene-dir>

uv run gauntlet realsim info <scene-dir>
```

`ingest` validates the frames + calibration JSON and writes a
self-contained scene directory (`manifest.json` + frame copies, or
symlinks via `--symlink`). The manifest carries `Pose` (4x4
row-major rigid transforms, NeRFStudio / COLMAP `transforms.json`
convention), `CameraIntrinsics` (pinhole + optional distortion,
shared by id), and `CameraFrame` rows. `info` prints a one-screen
manifest summary. The renderer itself is **deferred** — `RealSimRenderer`
is a `typing.Protocol`, and `register_renderer` / `get_renderer` are a
module-local registry for plugin renderers. See
[`docs/phase3-rfc-021-real-to-sim-stub.md`](./docs/phase3-rfc-021-real-to-sim-stub.md)
for the full design (pose representation, validation rules, plugin
seam).

### Multi-camera observations

Multi-view policies (SmolVLA, ACT, Diffusion Policy — anything that
consumes paired wrist + side + overhead frames) need more than the
single `obs["image"]` the legacy `render_in_obs=True` path emits.
Pass `cameras=[CameraSpec(...), ...]` to `TabletopEnv` or
`PyBulletTabletopEnv` and each spec lands in `obs["images"][name]`:

```python
from gauntlet.env import CameraSpec, TabletopEnv

env = TabletopEnv(
    cameras=[
        CameraSpec(name="wrist", pose=(0.0, 0.0, 0.4, 0.0, 0.0, 0.0), size=(96, 96)),
        CameraSpec(name="side",  pose=(0.5, 0.0, 0.3, 0.0, 1.2, 0.0), size=(96, 96)),
    ],
)
obs, _ = env.reset(seed=0)
wrist = obs["images"]["wrist"]  # shape (96, 96, 3), uint8
```

`CameraSpec.pose` is `(x, y, z, rx, ry, rz)` in metres + MuJoCo-XYZ
Euler radians (looks along local `-Z`); `CameraSpec.size` is `(H, W)`.
The legacy `obs["image"]` key stays populated as an alias to the
**first** camera's frame so single-view consumers (the runner's video
recorder, OpenVLA-style adapters) keep working unchanged. The
single-camera default (`cameras=None`) is byte-identical to the
phase-1 contract. See
[`docs/polish-exploration-multi-camera.md`](./docs/polish-exploration-multi-camera.md)
for the full design and
[`examples/evaluate_multi_camera.py`](./examples/evaluate_multi_camera.py)
for a worked example.

## Extending gauntlet

Third-party policies and envs plug into gauntlet through Python's
standard `importlib.metadata` entry-point mechanism — any
pip-installable package can register itself without modifying
gauntlet's source. Two groups are read by `gauntlet.plugins`:

| Entry-point group   | Registers                                                               |
|---------------------|-------------------------------------------------------------------------|
| `gauntlet.policies` | A class (or zero-arg callable) returning a `gauntlet.policy.base.Policy` |
| `gauntlet.envs`     | A class returning a `gauntlet.env.base.GauntletEnv`                      |

A plugin author writes the adapter, then declares it in their own
`pyproject.toml`:

```toml
[project.entry-points."gauntlet.policies"]
sb3 = "my_gauntlet_plugin.sb3_adapter:SBAdapter"
```

After `pip install my-gauntlet-plugin`, `gauntlet run ... --policy sb3`
resolves through the plugin path. Built-in adapters always win on
collision; failed entry-point loads are wrapped in a
`RuntimeWarning` and dropped from the registry — gauntlet itself
stays operational. See
[`docs/plugin-development.md`](./docs/plugin-development.md) for the
full how-to (writing a Policy / Env plugin, constructor-argument
patterns, testing) and
[`docs/polish-exploration-plugin-system.md`](./docs/polish-exploration-plugin-system.md)
for the design note (precedence rules, lazy discovery, collision
handling).

## Development

```bash
# Sync deps (creates .venv, installs everything in pyproject + dev group).
uv sync

# Lint, type-check, test.
uv run ruff check .
uv run mypy
uv run pytest
```

### Property tests

The `tests/test_property_*.py` and `tests/test_fuzz_*.py` files use
[Hypothesis](https://hypothesis.readthedocs.io) to fuzz invariants
that hand-rolled tests would only sample. They are tagged with the
`hypothesis_property` pytest marker so a focused run picks them up
without running the full suite:

```bash
# Run only the property/fuzz tests.
uv run pytest -m hypothesis_property

# Faster CI profile (max_examples=50 instead of the 200 default).
GAUNTLET_HYPOTHESIS_PROFILE=ci uv run pytest -m hypothesis_property
```

The covered invariants include:

* **Perturbation samplers** (`test_property_perturbation.py`,
  `test_property_axis_bounds.py`) — every emitted axis value lies
  inside the declared `[low, high]` envelope; same rng seed produces
  the same value.
* **Suite YAML round-trip** (`test_fuzz_suite_roundtrip.py`,
  `test_property_suite_loader.py`) — `Suite -> YAML -> Suite` is the
  identity for cartesian / LHS / Sobol suites; insertion order is
  preserved.
* **Action clipping** (`test_property_action_clipping.py`,
  `test_fuzz_action_space.py`) — finite-but-extreme action vectors
  produce per-step mocap deltas bounded by `MAX_LINEAR_STEP`. NaN/Inf
  inputs are pinned via `pytest.xfail` as a spec for future hardening.
* **Reset ordering** (`test_property_reset_after_step_ordering.py`)
  — `env.reset(S) -> step* -> env.reset(S)` produces bit-equal
  starting obs regardless of the intervening actions.
* **Observation NaN/Inf detection**
  (`test_property_observation_validation.py`) — spec-only via
  `pytest.xfail`; pins the gap that `Episode.observation_invalid`
  does not yet exist.

## Project layout

```
src/gauntlet/
  policy/      # Policy adapter protocol + reference wrappers (Random, Scripted, HF, LeRobot)
  env/         # Parameterized envs — MuJoCo (core) + PyBullet/Genesis/Isaac (extras)
  suite/       # YAML-defined perturbation grid suites (cartesian / LHS / Sobol)
  runner/      # Parallel rollout orchestration + seed management + cache
  report/      # Per-run failure analysis + HTML/JSON generation
  monitor/     # Runtime drift detection + action-entropy ([monitor] extra)
  replay/      # Single-episode replay with axis overrides
  ros2/        # ROS 2 publisher + recorder ([ros2] extra; rclpy via apt/Docker)
  diff/        # Structured per-axis report deltas powering `gauntlet diff`
  aggregate/   # Fleet-wide failure-mode clustering across many runs
  dashboard/   # Self-contained static SPA indexing every report.json
  realsim/     # Real-to-sim scene ingestion + RealSimRenderer Protocol (renderer deferred)
  plugins.py   # Entry-point discovery for third-party policies / envs
  cli.py       # gauntlet run / report / compare / diff / aggregate /
               # dashboard / realsim / monitor / replay / ros2
```

## License

MIT.
