Metadata-Version: 2.4
Name: cotterbot
Version: 0.3.0
Summary: Compliance-testing framework for AI-controlled robot policies — pytest for robot policies.
License-Expression: MIT
License-File: LICENSE
Author: yihan
Author-email: hyihan2007hong@gmail.com
Requires-Python: >=3.11,<3.12
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Provides-Extra: falsify
Provides-Extra: onnx
Provides-Extra: stl
Requires-Dist: cma (>=4.0,<5.0) ; extra == "falsify"
Requires-Dist: gymnasium-robotics (>=1.3,<2.0)
Requires-Dist: gymnasium[mujoco] (>=1.0,<2.0)
Requires-Dist: mujoco (>=3.2,<4.0)
Requires-Dist: numpy (>=1.26,<2.0)
Requires-Dist: onnxruntime (>=1.17,<2.0) ; extra == "onnx"
Requires-Dist: pyyaml (>=6.0,<7.0)
Requires-Dist: rtamt (>=0.3,<0.4) ; extra == "stl"
Requires-Dist: scipy (>=1.13,<2.0)
Requires-Dist: stable-baselines3 (>=2.4,<3.0)
Requires-Dist: torch (>=2.4,<3.0)
Description-Content-Type: text/markdown

# Cotter

**Compliance testing for AI-controlled robot policies — pytest for robot
policies.**

Cotter loads a trained robot policy as a black box (observation → action)
— a Stable-Baselines3 `.zip`, a raw PyTorch `.pt`/`.pth` module, or an
ONNX `.onnx` model — runs it through a battery of standardized tests in
MuJoCo simulation, and produces structured pass/fail results with
statistical guarantees. It is
aimed at the emerging regulatory need (EU Machinery Regulation, ISO 10218)
for evidence that a learned controller actually behaves — but the core is
just honest, reproducible testing:

| Category | Question | Method |
|---|---|---|
| **Performance** | Does it succeed at the task? | Wald's sequential probability ratio test (SPRT) — stops sampling as soon as the evidence is decisive |
| **Safety** | Does it ever exceed a hard physical limit? | Per-timestep checks on joint velocities / actuator forces / contacts; a single violation anywhere fails, no averaging. Includes an **ISO/TS 15066** power-and-force-limiting check (per-body-region contact-speed limits) |
| **Regression** | Did the new version break behavior? | Matched pairs on a shared seed sequence, exact McNemar (binary) and Wilcoxon signed-rank (continuous) |
| **Adversarial** | How bad is the worst case? | A PPO adversary trained to perturb the policy's *observations* (sensor attack) or *actions* (actuator attack) within an L∞ budget, plus a guaranteed random-noise baseline |
| **Specification** | Does it meet a declared behavioral requirement? | Signal Temporal Logic (STL) specs scored by *quantitative robustness* per rollout — e.g. `always (speed <= 1.5)`, `eventually (dist_to_goal <= 0.05)` |

Everything runs on CPU (developed on Apple Silicon; no CUDA anywhere in
the stack).

## Install

Requires Python 3.11. From PyPI (the distribution is published as
`cotterbot`; the import name, CLI, and project are all still `cotter`):

```sh
pip install cotterbot
```

To load **ONNX** policies, install the optional extra (pulls in `onnxruntime`):

```sh
pip install "cotterbot[onnx]"
```

> **Note:** on some systems the `cotter` binary lands outside your PATH (pip
> will warn you with the exact directory). Either add that directory to PATH,
> or invoke via `python -m cotter` instead:
>
> ```sh
> python -m cotter run --policy policy.zip --config run.yaml
> ```

Or from source with [Poetry](https://python-poetry.org/):

```sh
git clone https://github.com/yih0nk/cotter.git
cd cotter
poetry install
poetry run pytest   # 435 tests, unit + real-MuJoCo integration
```

## Quickstart (CLI)

Declare the test battery in YAML and point `cotter run` at a policy:

```sh
poetry run cotter run \
    --policy artifacts/victim_ppo_inverted_pendulum.zip \
    --config examples/inverted_pendulum.yaml
```

Exit code 0 means every declared category passed; 1 means at least one
failed; 2 means a config/usage error. Real captured output from the
command above (2026-07-07, seed 0 — trial-by-trial progress lines
elided):

```
[cotter] loaded policy 'victim_ppo_inverted_pendulum' onto InvertedPendulum-v5
[cotter] performance: SPRT p0=0.8 p1=0.95 n_max=50
[cotter]   => PASS after 18 trials
[cotter] safety: 4 limit(s) over 20 episodes
[cotter]   worst |cotter/joint_velocities| = 0.6789 (limit 5.0)
[cotter]   worst |cotter/actuator_forces| = 0.9062 (limit 2.5)
[cotter]   worst |cotter/contact_count| = 0.0000 (limit 0.5)
[cotter]   worst |cotter/contact_forces| = 0.0000 (limit 1.0)
[cotter]   => PASS
[cotter] regression: vs baseline .../victim_ppo_inverted_pendulum.zip on 30 paired seeds
[cotter]   => McNemar NO_REGRESSION (p=1), Wilcoxon NO_REGRESSION (p=1)
[cotter] adversarial: eps=0.07 over 20 episodes
[cotter]   random baseline: 100% clean -> 100% perturbed
[cotter]   ppo adversary: 100% clean -> 0% perturbed
[cotter] JSON report written to .../artifacts/cli_report.json
...
OVERALL: FAIL (1 failing, 5 passing, 0 informational)
```

(The FAIL is the learned-adversary category doing its job; the config's
regression section compares the victim against itself as a sanity check,
hence p = 1.) The config file schema is documented in
[`examples/inverted_pendulum.yaml`](examples/inverted_pendulum.yaml) and
`cotter/config.py`. Every reported success rate carries a Clopper-Pearson
confidence interval (SPRT and adversarial results), and regression
results carry an effect size — the discordant-pair odds ratio for
McNemar, the rank-biserial correlation for Wilcoxon.

#### Adversarial attack surface

The adversarial category attacks either surface, set by
`adversarial.attack`:

- `observation` (default) — perturb what the victim *sees* (sensor noise,
  calibration drift).
- `action` — perturb what the victim *does* (actuator noise, control-channel
  corruption), clipped back into the action space.
- `both` — run both; the report carries distinct `action_*` entries.

Each surface runs a guaranteed random baseline and, when `train: true`, a
PPO adversary trained against the frozen victim.

### Specifications (Signal Temporal Logic)

Beyond the fixed categories, declare behavioral requirements in **Signal
Temporal Logic** and Cotter scores each rollout by *quantitative
robustness* — a signed margin (positive = satisfied, negative = violated,
and by how much), not just pass/fail. Requires the `stl` extra
(`pip install cotterbot[stl]`):

```yaml
stl:
  n_episodes: 20
  variables:
    speed: cotter/tcp_speed                          # scalar info key
    jv0:   {key: cotter/joint_velocities, index: 0}  # a vector component
  threshold: 0.0            # min robustness required to pass (0 = the STL boundary)
  specs:
    - name: speed_cap
      formula: "always (speed <= 1.5)"
    - name: settles
      formula: "eventually (speed <= 0.1)"
```

Each spec becomes a `specification`-category result reporting the worst
(minimum) episode robustness. Signals are pulled from the per-step `info`
dict by the `variables` map. STL operators (`always`, `eventually`,
`until`, bounded `[a,b]`) are evaluated offline with
[`rtamt`](https://github.com/nickovic/rtamt).

### Falsification (active worst-case search)

The categories above *sample* scenarios; falsification *searches* for the
worst one. Point CMA-ES at an STL-robustness objective and it hunts the
initial condition / disturbance / parameter vector that most violates a
spec — returning the concrete counterexample. Requires the `falsify`
extra (`pip install cotterbot[falsify]`); a Python-API feature, since the
scenario→env mapping is code:

```python
import cotter

# how a scenario vector configures the env (masses, frictions, disturbance...)
def apply_scenario(env, params):
    env.unwrapped.model.body_mass[1] = params[0]      # e.g. sweep a link mass

objective = cotter.stl_scenario_objective(
    policy, env_factory, apply_scenario,
    formula="always (tcp_speed <= 1.5)",
    variables={"tcp_speed": {"key": "cotter/tcp_speed"}},
)
result = cotter.falsify(objective, lower=[0.5], upper=[5.0], max_evaluations=200)
if result.falsified:
    print("counterexample scenario:", result.best_params)
```

`falsify` minimizes the objective and reports whether it dropped below the
threshold (0 = a spec violation). Derivative-free and CPU-only — each
evaluation is one rollout; 1-D spaces use a deterministic sweep, higher
dimensions use CMA-ES.

### Coverage sweeps (operating-envelope coverage)

A green report shouldn't be able to hide untested regions. Coverage
sweeps the scenario space with a low-discrepancy **Sobol** sequence and
reports the metric *as a function of the scenario* — the failure fraction
and worst region, not one aggregate number. Same `objective` as
falsification (sweep it here, search it there), CPU-only, no extra:

```python
import cotter

result = cotter.sweep(
    objective,                       # e.g. cotter.stl_scenario_objective(...)
    lower=[0.5, 0.0], upper=[5.0, 1.0],   # the operating envelope
    n_scenarios=64,
)
print(result.summary())              # "covered 64 scenarios, 7 below threshold (11% failure region) ..."
for o in result.outcomes:            # every (scenario, value, pass/fail) — a coverage map
    ...
```

Coverage-of-operating-envelope reporting is something regulators ask for
(UNECE-style ODD coverage) but no open policy-eval tool provides.

For the common case — coverage across the **observation-disturbance**
envelope — a `coverage` config section runs it as part of `cotter run`
(no code), sweeping a constant observation offset in the L∞ box and
reporting a `coverage`-category result:

```yaml
coverage:
  epsilon: 0.1          # L∞ budget on the observation offset
  n_scenarios: 32
  n_episodes: 10
  min_success_rate: 0.8  # a scenario fails below this
```

### ISO/TS 15066 power-and-force-limiting (PFL)

Collaborative-robot safety comes down to a concrete numeric question: at
its speed, would a contact exceed the human injury threshold for the body
region it might hit? Cotter checks it directly. Declare an
`iso_ts_15066` section:

```yaml
iso_ts_15066:
  m_robot: 5.0                 # effective robot mass at the TCP (kg), incl. payload
  speed_key: cotter/tcp_speed  # per-step info key holding TCP linear velocity
  contact_type: transient      # transient (dynamic) or quasi_static (clamping)
  regions: [chest, hand_finger, skull_forehead]   # omit for the full library
```

For each region, Cotter back-solves the maximum permissible contact speed
from the ISO/TS 15066 collision model — `F = v·√(k·μ)`, reduced mass
`1/μ = 1/m_robot + 1/m_human` — and flags any timestep whose TCP speed
would produce a force over that region's limit, reporting the binding
(most restrictive) region and the implied contact force.

> The bundled per-region limits (`cotter.ISO_TS_15066_LIMITS`) are the
> widely-published approximations and are **user-editable** — confirm them
> against the purchased ISO/TS 15066 before relying on them for
> certification.

### Clause traceability

Conformity assessment is clause-by-clause. Declare which checks evidence
each regulatory clause and Cotter reports every clause as verified,
failed, or **unverified** — the coverage-gap view an auditor asks for:

```yaml
traceability:
  EHSR-1.3.7:
    description: "moving parts of machinery"
    checks: [hard_limits, iso_ts_15066_pfl]
  AI-Act-15:
    description: "robustness & accuracy"
    checks: [learned_ppo, speed_cap]
```

A clause is *failed* if any of its checks failed, *verified* if it has a
passing check and none failed, else *unverified* (no evidence). The result
is a `traceability`-category entry that fails the report if any clause is
failed or unverified. (The certified clause *thresholds* are the paid
layer; this open mapping ties your checks to the clauses you declare.)

### Reports (JSON + HTML + JUnit + Markdown)

Every run prints a console summary and can write four artifacts:

- **JSON** (`report:` in the config, or `--report`) — the machine-readable
  evidence, one object per category with the full statistics.
- **HTML** (`report_html:` in the config, or `--report-html`) — a single
  self-contained page you can open in a browser, drop into a pull request,
  or attach to a demo. No external assets, no JavaScript, adapts to the
  viewer's light/dark theme.
- **JUnit XML** (`report_junit:` in the config, or `--report-junit`) — each
  category as a `<testcase>`, so a Cotter run shows up as native test
  results in GitHub Actions, GitLab, Jenkins, CircleCI, etc.
- **Markdown** (`report_md:` in the config, or `--report-md`) — a shareable
  plain-text report to paste into a PR, an issue, or a wiki.

```sh
poetry run cotter run \
    --policy artifacts/victim_ppo_inverted_pendulum.zip \
    --config examples/inverted_pendulum.yaml \
    --report report.json --report-html report.html
```

The HTML page is the *free* counterpart to the paid compliance layer: the
open engine renders the pass/fail evidence into something shareable; the
licensed layer renders it into a signed, regulator-ready technical file.

### Reproducibility manifest

Every report (schema v2+) carries a `manifest` recording the full
provenance of the run — so a result can be reproduced and audited, not
just read:

```json
"manifest": {
  "cotter_version": "0.2.0",
  "python_version": "3.11.9",
  "platform": "Linux-6.5.0-x86_64",
  "dependencies": { "gymnasium": "1.0.0", "mujoco": "3.2.3", "torch": "2.4.1", ... },
  "env_id": "InvertedPendulum-v5",
  "base_seed": 0,
  "policy_path": "artifacts/victim.zip",
  "policy_sha256": "sha256:1f3a…"
}
```

The report also includes `content_sha256`, a digest over the report body
(excluding the timestamp and the digest itself). It is a stable id and a
tamper-evidence check.

Verify a report's integrity with `cotter verify`:

```sh
cotter verify report.json                 # recompute + check content_sha256
cotter verify report.json --policy p.zip  # also re-hash the policy vs the manifest
```

Exit 0 means verified, 1 means a check failed (the report or policy was
modified), 2 means the report is missing or invalid.

### Comparing two policy versions

`cotter compare` runs only the regression test between a baseline and a
candidate and exits 0 (no regression) or 1 (regression detected) — a
drop-in CI gate:

```sh
poetry run cotter compare \
    --baseline old_policy.zip --candidate new_policy.zip \
    --config examples/inverted_pendulum.yaml
```

Like `run`, it accepts `--report` (JSON) and `--report-html` to write the
comparison result to disk — handy for archiving the artifact or feeding a
PR comment.

### Diffing reports (safety budget)

Where `compare` re-runs the regression test, `cotter diff` compares two
existing **reports** and fails if any previously-passing check now fails —
a safety-budget gate across the *whole* battery, not just regression:

```sh
cotter diff baseline_report.json candidate_report.json
```

Each check is classified regressed / improved / added / removed; exit 1
means a check that used to pass now fails (0 otherwise, 2 on a bad
report). Store a baseline report and diff every retrain against it.

### Parallel rollouts

Safety, regression, and fixed-N adversarial evaluation parallelize across
worker processes via `AsyncVectorEnv`. Set `n_workers` in the relevant
config section (default 1 = serial); SPRT stays serial by design because
it must check its boundary after each trial. Parallel results are
bit-identical to serial on the same seeds — each episode runs in an
isolated env seeded from a disjoint per-worker slice.

```yaml
safety:
  n_episodes: 40
  n_workers: 4
  limits: { cotter/joint_velocities: 5.0 }
```

### Managing the adversary zoo

```sh
poetry run cotter zoo list                 # cached adversaries (+ --env filter)
poetry run cotter zoo prune                # drop entries with missing artifacts
```

### Backends

`RunConfig.backend` (default `gymnasium`) selects the simulator backend
through `BackendFactory.from_name`. `GymnasiumBackend` is the CPU MuJoCo
default; `IsaacSimBackend` targets NVIDIA Isaac Sim and raises
`BackendNotAvailableError` at construction when `omni.isaac.gym` is not
installed (it is not exercised on the CPU dev machine).

## Quickstart (Python API)

```python
import gymnasium as gym
from stable_baselines3 import PPO
from cotter import (
    CotterWrapper, SafetyLimit, TestReport, load_policy, run_rollouts,
    run_sprt, evaluate_safety, mcnemar_exact, run_adversarial_test,
    rollout_one, make_seed_sequence, JOINT_VELOCITIES, ACTUATOR_FORCES,
)

# 1. Wrap any Gymnasium MuJoCo env; the wrapper exposes qvel,
#    actuator_force, contact count, and per-body contact-force
#    magnitudes in every step's info dict.
env = CotterWrapper(gym.make("InvertedPendulum-v5"))

# 2. Load the policy under test (SB3 .zip, raw torch .pt, or .onnx).
#    Observation/action spaces are validated and mismatches fail loudly.
policy = load_policy("artifacts/victim_ppo_inverted_pendulum.zip", env, algo=PPO)

# 3. Define task success and run the categories you need.
def success(total_reward, length, terminated, truncated, final_info):
    return length >= 1000  # survived the full horizon

seeds = make_seed_sequence(50, base_seed=0)
perf = run_sprt(
    lambda i: rollout_one(policy, env, seeds[i], success).success,
    p0=0.80, p1=0.95, alpha=0.05, beta=0.05, n_max=50,
)

rollouts = run_rollouts(policy, env, 20, success, base_seed=1)
safe = evaluate_safety(rollouts.episode_infos, [
    SafetyLimit(JOINT_VELOCITIES, 5.0),
    SafetyLimit(ACTUATOR_FORCES, 2.5),
])

adv = run_adversarial_test(policy, env, success, epsilon=0.07, n_episodes=20)

# 4. Aggregate into a report (console summary + JSON + HTML artifacts).
report = TestReport(policy_name="my_policy", env_id="InvertedPendulum-v5")
report.add_sprt(perf)
report.add_safety(safe)
report.add_adversarial(adv)
print(report.summary())
report.to_json("report.json")
report.to_html("report.html")   # self-contained, shareable page
```

## Demo

```sh
poetry run python examples/demo.py
```

Runs all four categories against a checked-in PPO policy and writes
`artifacts/demo_report.json`. Takes ~40 s on Apple Silicon CPU (dominated
by training the adversary; pass `--skip-adversary-training` to use only
the random baseline). The victim can be retrained from scratch with
`poetry run python scripts/train_victim.py` (~17 s, 100k timesteps,
eval reward 1000.0 ± 0.0).

### Demo environment choice

The demo uses **InvertedPendulum-v5**, chosen deliberately for
reliability over spectacle: it steps fast on CPU, PPO solves it in
seconds (so the whole pipeline is verifiable end-to-end in one sitting),
and its MuJoCo `data` exposes physically meaningful joint velocities,
actuator forces, and contact counts for the safety checks. Nothing in
Cotter is specific to this env — `CotterWrapper` works with any
Gymnasium MuJoCo environment with Box spaces.

### Real captured results

Verbatim summary from an executed run (2026-07-07, seed 0, M5 CPU —
full output in [RESULTS.md](RESULTS.md)):

```
==============================================================================
COTTER TEST REPORT — policy 'victim_ppo' on InvertedPendulum-v5
generated 2026-07-07T12:54:58+00:00
==============================================================================
[PASS] performance/sprt_success_rate
       PASS: 18/18 successes (100.0%) after 18 sequential trials (H0 p<=0.8, H1 p>=0.95)
[PASS] safety/hard_limits
       PASS: no violations in 20020 timesteps across 20 trials
[FAIL] regression/success_mcnemar
       REGRESSION: baseline 1.000 vs candidate 0.000 over 30 paired trials (p=9.31e-10, mcnemar_exact_one_sided)
[FAIL] regression/return_wilcoxon
       REGRESSION: baseline 1000.000 vs candidate 94.900 over 30 paired trials (p=8.63e-07, wilcoxon_signed_rank_one_sided)
[PASS] adversarial/random_baseline
       PASS: success rate 100.0% clean -> 100.0% under random linf perturbation (eps=0.07, n=20, required >= 50%) [uniform random baseline]
[FAIL] adversarial/learned_ppo
       FAIL: success rate 100.0% clean -> 0.0% under ppo linf perturbation (eps=0.07, n=20, required >= 50%) [trained PPO adversary]
------------------------------------------------------------------------------
OVERALL: FAIL (3 failing, 3 passing, 0 informational)
==============================================================================
```

The FAILs are the point of the demo: the regression category is fed a
deliberately undertrained candidate and catches it (p ≈ 10⁻⁹ from just
30 paired episodes), and the adversarial category shows the headline
result — **at a perturbation budget where random sensor noise is
completely harmless (100% success), a learned adversary drives the same
policy to 0%.** Random-noise robustness testing alone would have
certified this policy.

## Locomotion demo (Ant-v5)

A PPO victim trained on the MuJoCo quadruped (500k steps,
`scripts/train_ant_victim.py`), tested with all four categories via
`cotter run --policy artifacts/victim_ant.zip --config examples/ant.yaml`.
Real captured report (2026-07-08, seed 0, M5 CPU):

```
COTTER TEST REPORT — policy 'victim_ant' on Ant-v5
[PASS] performance/sprt_success_rate
       PASS: 7/7 successes (100.0%, 95% CI [59.0%, 100.0%]) after 7 sequential trials (H0 p<=0.5, H1 p>=0.8)
[PASS] safety/hard_limits
       PASS: no violations in 12813 timesteps across 20 trials
[PASS] regression/success_mcnemar
       NO_REGRESSION: baseline 0.000 vs candidate 0.950 over 20 paired trials (p=1, discordant_odds_ratio=0, ...)
[PASS] regression/x_position_wilcoxon
       NO_REGRESSION: baseline -0.066 vs candidate 9.446 over 20 paired trials (p=1, rank_biserial_correlation=0.971, ...)
[PASS] adversarial/random_baseline
       PASS: success rate 100.0% clean -> 75.0% [50.9%, 91.3%] under random linf perturbation (eps=1.0, n=20, required >= 50%)
[PASS] adversarial/learned_ppo
       PASS: success rate 100.0% clean -> 95.0% [75.1%, 99.9%] under ppo linf perturbation (eps=1.0, n=20, required >= 50%)
OVERALL: PASS (0 failing, 6 passing, 0 informational)
```

Locomotion needs the right success signal: Ant's per-step *survival
bonus* means a policy that stands still scores high total reward, so raw
return would rank a do-nothing policy above a walking one. The demo
therefore defines success as **forward displacement** (`min_info` on
`x_position`) and runs the regression on the same metric. That cleanly
separates the trained victim (95% success, ~9.4 m forward) from the
random-init baseline (0%, ~0 m). Here the learned adversary happened to
underperform the random baseline at this budget (the PPO attacker is
time-boxed); Cotter reports both honestly.

## Real manipulation demo (HER + SAC)

Cotter handles the `Dict` observation spaces used by
[gymnasium-robotics](https://robotics.farama.org/) Fetch/manipulation
envs, where success is a goal flag (`{type: info_flag, key: is_success}`)
rather than a reward threshold.

The intended target was **FetchPickAndPlace-v4** trained with HER+SAC
(`scripts/train_fetch_pickplace.py`), but it did not converge on the M5
CPU within the time budget (0% success through 100k steps). Per the
documented fallback the demo uses the much easier **FetchReachDense-v4**,
which the same HER+SAC recipe solves to 100% in about a minute. Real
captured report from `cotter run --policy
artifacts/victim_fetch_reach_hersac.zip --config
examples/fetch_pickplace.yaml`:

```
COTTER TEST REPORT — policy 'victim_fetch_reach_hersac' on FetchReachDense-v4
[PASS] performance/sprt_success_rate
       PASS: 8/8 successes (100.0%, 95% CI [63.1%, 100.0%]) after 8 sequential trials (H0 p<=0.4, H1 p>=0.6)
[PASS] safety/hard_limits
       PASS: no violations in 1020 timesteps across 20 trials
[PASS] regression/success_mcnemar
       NO_REGRESSION: baseline 0.033 vs candidate 1.000 over 30 paired trials (p=1, discordant_odds_ratio=0, ...)
[PASS] regression/return_wilcoxon
       NO_REGRESSION: baseline -8.198 vs candidate -1.013 over 30 paired trials (p=1, rank_biserial_correlation=1, ...)
[PASS] adversarial/random_baseline
       PASS: success rate 100.0% clean -> 55.0% [31.5%, 76.9%] under random linf perturbation (eps=0.1, n=20)
[FAIL] adversarial/learned_ppo
       FAIL: success rate 100.0% clean -> 0.0% [0.0%, 16.8%] under ppo linf perturbation (eps=0.1, n=20, required >= 50%)
OVERALL: FAIL (1 failing, 5 passing, 0 informational)
```

The safety limits target Fetch's joint velocities and contacts (its
end-effector is mocap-controlled, so `actuator_forces` is empty and a
limit on it passes trivially). **Adversarial perturbation works on Dict
observations**: it perturbs only the sensed-state `observation` sub-key
(leaving the goal keys untouched), and the headline result reappears on
a manipulation env — the learned PPO adversary drives the victim to 0%
at ε = 0.1 where random noise of the same budget only reaches 55%.

Regression *detection* is a one-liner. Comparing the trained victim
against its random-init counterpart flags the difference immediately:

```
$ cotter compare --baseline victim_fetch_reach_hersac.zip \
      --candidate victim_fetch_reach_randominit.zip --config examples/fetch_pickplace.yaml
[FAIL] regression/success_mcnemar
       REGRESSION: baseline 1.000 vs candidate 0.033 over 30 paired trials (p=1.86e-09, discordant_odds_ratio=inf, ...)
[FAIL] regression/return_wilcoxon
       REGRESSION: baseline -1.013 vs candidate -8.198 over 30 paired trials (p=9.31e-10, rank_biserial_correlation=-1, ...)
OVERALL: FAIL (2 failing, 0 passing, 0 informational)   # exit code 1
```

## Adversary zoo

Trained adversaries are expensive and specific to what they attack, so
`cotter.zoo.AdversaryZoo` caches them keyed by
`(env_id, victim_hash, epsilon)`. Set `use_zoo: true` in a config's
adversarial section and the first run trains and stores the attacker;
later runs on the same victim reuse it instead of retraining. The victim
hash is taken over the policy artifact (or its parameter tensors), so a
retrained victim never silently reuses a stale adversary.

### Pretrained adversaries (transfer attacks)

`cotter.zoo.PretrainedZoo` is a different registry: keyed by **robot
class** — `(robot_class, env_id, epsilon, attack)` — not by victim. An
adversary trained per class transfers to *any* compatible victim on that
env (adversarial policies exploit the environment, not victim-specific
weights), so you can attack your own policy without training one:

```yaml
adversarial:
  epsilon: 0.05
  pretrained: franka-panda    # transfer-attack from the pretrained zoo
```

The run loads the class adversary, checks it matches the env, and reports
a `pretrained_<class>_<surface>` category (a missing class or incompatible
adversary is skipped, never fatal). Inspect the registry with
`cotter pretrained list`. The open engine ships the mechanism
(`register` / `transfer_attack`); a hosted library of GPU-trained experts
per robot class is the paid tier.

## Compliance layer (paid tier)

The open-source engine produces the *evidence* — structured pass/fail
results with statistical guarantees. Rendering that into regulator-ready
documentation (EU Machinery Regulation 2027 technical files, ISO 10218
conformity records) is the licensed commercial layer, stubbed here with
a stable import path:

```python
from cotter.compliance import EUMachineryReg2027
EUMachineryReg2027(report)   # raises LicenseRequiredError in the OSS build
```

## Architecture

```
cotter/
├── cli.py               # `cotter run` / `compare` / `zoo` entrypoints
├── config.py            # YAML config schema + loader
├── pipeline.py          # executes the declared categories, aggregates a report
├── backends.py          # simulator backend abstraction (gymnasium / isaac-sim)
├── success.py           # declarative success criteria (min_length/return/info_flag)
├── stats.py             # Clopper-Pearson exact binomial confidence intervals
├── policy.py            # black-box loading (SB3 .zip / torch .pt) + space validation
├── runner.py            # seeded rollouts (serial + parallel) -> EpisodeRecords
├── envs/
│   ├── wrapper.py       # instruments info dict with qvel / actuator_force / contacts
│   ├── registry.py      # resolves gymnasium-robotics (and other) extension envs
│   └── factory.py       # picklable env factory for parallel rollouts
├── report.py            # TestReport: console summary + JSON (results container only)
├── render.py            # self-contained HTML rendering of a TestReport (free tier)
├── zoo/                 # adversary registries: victim-keyed cache + class-keyed pretrained
├── compliance/          # paid-tier regulatory document stub (license required)
└── tests/
    ├── sprt.py          # Wald's sequential probability ratio test (+ CI)
    ├── safety.py        # per-timestep hard limits, zero tolerance
    ├── regression.py    # exact McNemar + Wilcoxon on matched pairs (+ effect sizes)
    └── adversarial.py   # observation-perturbation attack (PPO or random, + CI)
```

Design notes:

- **Space validation fails loudly.** Loading a policy checks its declared
  and functional observation/action shapes against the env — the #1
  real-world integration bug surfaces at load time, not mid-rollout.
- **Shared seeds everywhere it matters.** Regression pairs and
  clean-vs-attacked comparisons run on identical seed sequences, so
  differences are attributable to the policy, not the physics draw.
- **The adversarial floor never fails.** If PPO adversary training
  errors, `get_adversary` falls back to the random baseline and says so
  in the result — the category always produces a number.
- **`report.py` is a results container**, not a compliance-document
  generator; that is the paid `cotter.compliance` layer.
- **Config-driven, seed-reproducible runs.** `cotter run --config
  run.yaml` executes whichever categories the YAML declares; every
  category derives its seeds from one `base_seed`, so enabling or
  disabling a category never perturbs another's rollouts.

