Metadata-Version: 2.4
Name: road-rl
Version: 0.0.2
Summary: RoAd-RL - Robust Adversarial Reinforcement Learning
Author-email: Adithya Mohan <herrmohan1394@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/codewiz1394/road-rl
Project-URL: Repository, https://github.com/codewiz1394/road-rl
Project-URL: Issues, https://github.com/codewiz1394/road-rl/issues
Keywords: reinforcement-learning,adversarial-attacks,adversarial-robustness,deep-learning,gymnasium
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: matplotlib>=3.7
Provides-Extra: rl
Requires-Dist: gymnasium>=0.28; extra == "rl"
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == "torch"
Provides-Extra: sb3
Requires-Dist: stable-baselines3>=2.3; extra == "sb3"
Provides-Extra: box2d
Requires-Dist: gymnasium[box2d]>=0.28; extra == "box2d"
Provides-Extra: plots
Requires-Dist: scienceplots>=2.1; extra == "plots"
Provides-Extra: config
Requires-Dist: pyyaml>=6.0; extra == "config"
Provides-Extra: dev
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

<p align="center">
  <img src="https://raw.githubusercontent.com/codewiz1394/road-rl/main/assets/logo.png" alt="RoAd-RL logo" width="300"/>
</p>

# RoAd-RL: Robust Adversarial Deep Reinforcement Learning

A Python library for benchmarking adversarial attacks and defenses in deep reinforcement learning. RoAd-RL provides a modular, reproducible pipeline for training RL agents, evaluating them under observation-space adversarial attacks, and applying inference-time defenses.

---

## Installation

```bash
pip install road-rl
```

Install optional extras based on what you need:

```bash
pip install "road-rl[torch]"       # PyTorch support
pip install "road-rl[sb3]"         # Stable-Baselines3
pip install "road-rl[rl]"          # Gymnasium
pip install "road-rl[box2d]"       # Box2D environments (LunarLander etc.)
pip install "road-rl[plots]"       # scienceplots for publication figures
pip install "road-rl[config]"      # YAML config support
```

Or install everything at once:

```bash
pip install -r requirements.txt
```

---

## Library Overview

RoAd-RL evaluates robustness by running episodes under a sweep of attack budgets (epsilon values) and seeds. At each step:

```
observation → [attack] → [defense] → policy → action
```

**Attacks (observation-space, gradient-based):**

| Name | Description | Norm |
|------|-------------|------|
| `fgsm` | Fast Gradient Sign Method — single-step | Linf, L2 |
| `pgd` | Projected Gradient Descent — iterative, optional random start | Linf, L2 |
| `jsma` | Jacobian Saliency Map Attack — perturbs top-k features | Linf |

**Defenses (inference-time preprocessing):**

| Name | Description |
|------|-------------|
| `normalize_clip` | Normalize then clip to observation bounds |
| `smoothing` | Moving-average temporal smoothing over a window |
| `gaussian_noise` | Randomized smoothing (adds Gaussian noise) |
| `feature_squeeze` | Quantize observations to N bits |
| `median_smoothing` | Median filter over observation window |
| `outlier_clip` | Clip outlier values beyond N standard deviations |
| `pca` | PCA projection for dimensionality-based denoising |

**Supported environments:**

| Environment | Adapter | Algorithms |
|---|---|---|
| `LunarLander-v2` | `gym` | DQN, PPO |
| `LunarLanderContinuous-v2` | `gym` | SAC |
| `highway-v0` | `highway` | DQN, PPO, SAC |
| `CartPole-v1`, `BipedalWalker-v3`, etc. | `gym` | DQN, PPO, SAC |
| Atari | `atari` | DQN |

---

## Repository Structure

```
road-rl/
├── road_rl/                  # Library package
│   ├── attacks/              # FGSM, PGD, JSMA adversarial attacks
│   ├── defenses/             # Normalize/clip, smoothing, gaussian noise, feature squeeze, PCA
│   ├── policies/             # Policy adapters: DQN, PPO, SAC, SB3
│   ├── envs/                 # Environment adapters: Gym, Highway, Atari
│   ├── eval/                 # Episode runner, sweep runner, evaluator
│   ├── metrics/              # Robustness, return, risk, safety metrics
│   ├── adv_training/         # Adversarial training wrappers and schedulers
│   ├── train/                # DQN/PPO/SAC trainers via Stable-Baselines3
│   ├── io/                   # Config loader, CSV/JSON logger, plotting
│   ├── utils/                # Training curve aggregation and plotting
│   └── cli/                  # `road-rl eval` CLI entry point
│
├── scripts/
│   ├── train_all.py                  # Batch training
│   ├── eval_all.py                   # Batch evaluation
│   ├── run_eval.py                   # Sequential or tmux sweep runner
│   ├── eval_sweep.py                 # Single sweep evaluation
│   ├── eval_rewards_only.py          # Clean reward evaluation helper
│   ├── eval_highway_vanilla.py       # Highway vanilla combined eval
│   ├── eval_highway_attacks.py       # Highway under attacks combined eval
│   ├── plot_all.py                   # Aggregate plots from episode CSVs
│   ├── plot_training_benchmarks.py   # Training curve benchmark plots
│   ├── highway_{dqn,ppo,sac}_main.py # Highway training entrypoints
│   ├── lunar_{dqn,ppo,sac}_main.py   # Lunar training entrypoints
│   └── release_check.py              # Pre-release validation
│
└── tests/                    # Unit tests
```

---

## Quick Start

### Programmatic sweep

```python
from road_rl.eval.sweep_runner import run_sweep
from road_rl.envs.make_env import EnvSpec, make_env_factory
from road_rl.train.utils import build_attack, build_defense, load_policy_from_checkpoint

env_factory = make_env_factory(EnvSpec("LunarLander-v2", adapter="gym"))
policy = load_policy_from_checkpoint("checkpoints/lunar_dqn.zip", algorithm="dqn")
attack = build_attack("fgsm", norm="linf")
defense = build_defense("normalize_clip")

result = run_sweep(
    env_factory=env_factory,
    policy=policy,
    env_id="LunarLander-v2",
    algorithm="dqn",
    epsilons=[0.0, 0.01, 0.05],
    seeds=[0, 1, 2],
    attack=attack,
    defense=defense,
    episodes_per_seed=30,
    show_progress=True,
)
# result.episodes → list of EpisodeResult
```

### CLI evaluation

```bash
road-rl eval \
  --env-id LunarLander-v2 \
  --adapter gym \
  --policy-path checkpoints/lunar_dqn.zip \
  --algorithm dqn \
  --eps 0.0 0.005 0.01 0.025 0.05 \
  --seeds 0 1 2 \
  --episodes-per-seed 40 \
  --attack fgsm \
  --defense none \
  --out results/
```

### Training agents

```bash
# Train a specific environment
python scripts/lunar_dqn_main.py
python scripts/highway_ppo_main.py

# Batch train from configs
python scripts/train_all.py
```

---

## Extending the Library

**Custom policy:**

```python
from road_rl.policies.base import Policy
import numpy as np

class MyPolicy(Policy):
    def act(self, obs: np.ndarray) -> int:
        ...
    def loss(self, obs_tensor) -> "torch.Tensor":
        ...  # differentiable loss for gradient-based attacks
```

**Custom attack:**

```python
from road_rl.attacks.base import Attack
from road_rl.core.context import StepContext

class MyAttack(Attack):
    def apply(self, obs, policy, ctx: StepContext):
        ...
```

**Custom defense:**

```python
from road_rl.defenses.base import Defense

class MyDefense(Defense):
    def apply(self, obs, ctx):
        ...
```

---

## License

MIT — see [LICENSE](LICENSE).
