Metadata-Version: 2.4
Name: neatrl
Version: 2.5.2
Summary: A Python library for reinforcement learning algorithms
Author-email: Yuvraj Singh <yuvraj.mist@gmail.com>
License: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-DATASET
License-File: NOTICE
Requires-Dist: torch
Requires-Dist: numpy
Requires-Dist: gymnasium[classic-control]>=1.3.0
Requires-Dist: pygame-ce>=2.1.3
Requires-Dist: stable-baselines3
Requires-Dist: tqdm
Requires-Dist: wandb[media]
Requires-Dist: imageio[ffmpeg]
Requires-Dist: opencv-python
Requires-Dist: rich
Requires-Dist: psutil
Requires-Dist: nvidia-ml-py
Provides-Extra: atari
Requires-Dist: gymnasium[atari]; extra == "atari"
Requires-Dist: ale-py; extra == "atari"
Provides-Extra: ale
Requires-Dist: gymnasium[atari]; extra == "ale"
Requires-Dist: ale-py; extra == "ale"
Provides-Extra: box2d
Requires-Dist: gymnasium[box2d]; extra == "box2d"
Provides-Extra: mujoco
Requires-Dist: gymnasium[mujoco]; extra == "mujoco"
Provides-Extra: cli
Requires-Dist: rich; extra == "cli"
Requires-Dist: psutil; extra == "cli"
Requires-Dist: nvidia-ml-py; extra == "cli"
Provides-Extra: classic
Requires-Dist: gymnasium[classic-control]; extra == "classic"
Provides-Extra: all
Requires-Dist: rich; extra == "all"
Requires-Dist: psutil; extra == "all"
Requires-Dist: nvidia-ml-py; extra == "all"
Requires-Dist: ale-py; extra == "all"
Requires-Dist: gymnasium[atari,box2d,classic-control,mujoco]; extra == "all"
Provides-Extra: dev
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: rich; extra == "dev"
Requires-Dist: psutil; extra == "dev"
Requires-Dist: nvidia-ml-py; extra == "dev"
Dynamic: license-file

<div align="center">

# NeatRL

**A clean Python library for reinforcement learning algorithms**

[![license: Apache 2.0](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE) [![PyPI](https://img.shields.io/pypi/v/neatrl)](https://pypi.org/project/neatrl/) [![dataset: CC BY 4.0](https://img.shields.io/badge/dataset-CC_BY_4.0-lightgrey)](LICENSE-DATASET) [![Sponsor: GitHub](https://img.shields.io/badge/Sponsor-GitHub-ea4aaa?logo=githubsponsors&logoColor=white)](https://github.com/sponsors/YuvrajSingh-mist) [![Buy me a coffee](https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=black)](https://ko-fi.com/O7W120DR8R)

</div>

NeatRL provides readable implementations of popular RL algorithms with a focus on simplicity and ease of use. Built with PyTorch and designed for research and experimentation.

## Features

- **Single-file implementations**: Each algorithm lives entirely in one self-contained file -- no hunting across modules
- **Experiment Tracking**: Built-in support for Weights & Biases logging
- **Gymnasium Compatible**: Works with Gymnasium environments and adding many more!
- **Atari Support**: Full support for Atari games with automatic CNN architectures
- **Parallel Training**: Vectorized environments for faster data collection
- **Easy to Extend**: Modular design for adding new algorithms
- **Video Recording**: Automatic video capture and WandB integration


## Installation

```bash
pip install neatrl
```

That is the whole default setup. It brings the terminal dashboard (`rich`,
`psutil`), classic-control environments (CartPole, Pendulum, Acrobot,
MountainCar) and everything the MLP aliases need — no extras to remember.

With [uv](https://docs.astral.sh/uv/), install the `neatrl` command as an
isolated tool — no virtualenv to manage:

```bash
uv tool install neatrl
```

### Image-based (`-cnn`) aliases

The `-cnn` aliases train on image observations, which only Atari and Box2D
provide — gymnasium's classic control and MuJoCo envs are all vector
observations, so there is no CNN equivalent for them. Add the family you want:

```bash
pip install "neatrl[atari]"    # ALE/Pong-v5, ALE/Breakout-v5 (default for -cnn)
pip install "neatrl[box2d]"    # CarRacing-v3, LunarLander-v3, BipedalWalker-v3
pip install "neatrl[mujoco]"   # HalfCheetah-v4, Ant-v5, …
pip install "neatrl[all]"      # all of the above in one go
```

If you run a `-cnn` alias without the matching extra, the CLI prints
gymnasium's error unchanged and then tells you the exact command to run.

`neatrl[cli]` and `neatrl[classic]` still resolve, so older instructions keep
working, but they are no longer needed.

> Browse supported environments: [MuJoCo](https://gymnasium.farama.org/environments/mujoco/) · [Box2D](https://gymnasium.farama.org/environments/box2d/) · [Classic Control](https://gymnasium.farama.org/environments/classic_control/) · [Atari](https://gymnasium.farama.org/environments/atari/)

## CLI

Train any algorithm from the terminal:

```bash
neatrl train dqn CartPole-v1
neatrl train ppo LunarLander-v3
neatrl train sac Pendulum-v1
neatrl train td3 HalfCheetah-v4
neatrl train rnd CliffWalking-v1
neatrl train ddpg-cnn CarRacing-v3
```

The environment is optional — omit it to use the algorithm's default
(`neatrl train dqn` trains on `CartPole-v1`, `neatrl train sac` on
`Pendulum-v1`). `--steps` overrides the training length:

```bash
neatrl train dqn CartPole-v1 --steps 50000
```

Runs are local: nothing is uploaded to Weights & Biases unless you pass
`--wandb`.

Pass `--help` to any command for the full list of the 21 algorithm aliases.


## Quick Start

### Train DQN on CartPole

```python
from neatrl.dqn_mlp import train_dqn

model = train_dqn(env_id="CartPole-v1", total_timesteps=10000, seed=42)
```

### Train tabular Q-learning on FrozenLake

```python
from neatrl.q_learning import train_q_learning

q_table = train_q_learning(
    env_id="FrozenLake-v1",
    total_timesteps=50000,
    seed=42,
    use_wandb=False,
)
```

Works on simple discrete Gym envs only (`FrozenLake`, `CliffWalking`, `Taxi`, …). For continuous observations use DQN.

### Train PPO on Classic Control

```python
from neatrl.ppo_mlp import train_ppo

model = train_ppo(
    env_id="CartPole-v1",
    total_timesteps=50000,
    n_envs=4,  # Parallel environments
    GAE=0.95,  # Generalized Advantage Estimation lambda
    clip_value=0.2,  # PPO clipping parameter
    use_wandb=True,  # Track with WandB
    seed=42,
)
```

### Train SAC on Continuous Control

```python
from neatrl.sac_mlp import train_sac

model = train_sac(
    env_id="Pendulum-v1",
    total_timesteps=50000,
    alpha=0.2,  # Entropy regularization coefficient
    autotune_alpha=True,  # Automatically tune alpha
    use_wandb=True,  # Track with WandB
    seed=42,
)
```

### Train SAC on Atari

```python
from neatrl.sac_cnn import train_sac_cnn

model = train_sac_cnn(
    env_id="ALE/Breakout-v5",
    total_timesteps=100000,
    alpha=0.2,
    autotune_alpha=True,
    atari_wrapper=True,  # Automatic Atari preprocessing
    use_wandb=True,
    seed=42,
)
```


## Documentation

**[Complete Documentation](https://github.com/YuvrajSingh-mist/NeatRL/tree/master/docs)**

The docs include:
- Detailed usage examples for every algorithm
- Hyperparameter tuning guides
- Environment compatibility notes
- Experiment tracking setup (WandB)


## Supported Algorithms

### Current Implementations

- **DQN** (Deep Q-Network) - Classic value-based RL algorithm
  - Support for discrete action spaces
  - Experience replay and target networks
  - Atari preprocessing and frame stacking

- **Dueling DQN** - Enhanced DQN with separate value and advantage streams
  - Improved learning stability
  - Better performance on complex environments

- **Double DQN** - Q-learning target with decoupled action selection/evaluation
  - Removes the upward bias of the `max` in the DQN target
  - Same API and hyperparameters as DQN (`double_dqn_mlp`, `double_dqn_cnn`)

- **REINFORCE** - Policy gradient method for discrete and continuous action spaces
  - Atari game support with automatic CNN architecture
  - Parallel environment training (`n_envs` support)
  - Continuous action space support
  - Episode-based Monte Carlo returns
  - Variance reduction through baseline subtraction

- **DDPG** (Deep Deterministic Policy Gradient) - Actor-critic method for continuous action spaces
  - Deterministic policy gradient for continuous control
  - Experience replay and target networks
  - Ornstein-Uhlenbeck noise for exploration
  - Support for exact continuous action spaces

- **A2C** (Advantage Actor-Critic) - Synchronous actor-critic algorithm
  - Advantage function for reduced variance
  - Support for both discrete and continuous action spaces
  - Parallel environment training with vectorized environments
  - Monte Carlo returns for value estimation

- **TD3** (Twin Delayed DDPG) - Actor-critic method for continuous control
  - Twin Q-networks to reduce overestimation bias
  - Delayed policy updates for improved stability
  - Target policy smoothing with noise
  - Experience replay and target networks
  - CNN support for image-based environments

- **SAC** (Soft Actor-Critic) - Maximum entropy reinforcement learning
  - Stochastic Gaussian policies with entropy regularization
  - Twin Q-networks for stable learning
  - Automatic entropy tuning (alpha parameter)
  - Balances exploration and exploitation
  - CNN support for complex environments

- **PPO** (Proximal Policy Optimization) - Policy gradient method with GAE
  - Full PPO implementation with Generalized Advantage Estimation (GAE)
  - Support for both discrete and continuous action spaces
  - Atari game support with automatic CNN architecture
  - Clipped surrogate objective for stable policy updates
  - Value function clipping and entropy regularization
  - Vectorized environments for parallel training

- **PPO-RND** (Proximal Policy Optimization with Random Network Distillation) - Curiosity-driven exploration
  - Intrinsic motivation through novelty detection
  - Combined extrinsic and intrinsic rewards for better exploration
  - Support for both discrete and continuous action spaces
  - PPO with clipped surrogate objective
  - Vectorized environments for parallel training
  - Intrinsic reward normalization and advantage calculation

- *More algorithms coming soon...*

## Contributing

Contributions are welcome. Please feel free to submit a Pull Request.

### Development Setup

```bash
git clone https://github.com/YuvrajSingh-mist/NeatRL.git
cd NeatRL/NeatRL

# Use a non-hidden venv/ (avoids a macOS/uv flag issue with .venv/)
uv venv venv
uv pip install -e ".[cli,dev]" --python venv/bin/python
source venv/bin/activate

pytest tests/           # run the test suite
ruff check src/         # lint
neatrl train ppo lunar   # smoke test CLI
```

> **When to re-run `uv pip install -e .`:** Only when you add/remove/rename a module file or change `pyproject.toml` dependencies. Editing existing `.py` files never requires a reinstall — the editable install picks up changes automatically.

For the complete changelog, see [CHANGELOG.md](https://github.com/YuvrajSingh-mist/NeatRL/blob/master/CHANGELOG.md).

## Citation

If you use `NeatRL` in academic work, please cite it as below. The concept DOI always resolves to the newest archived version (see [`CITATION.cff`](CITATION.cff) for the machine-readable form).

```bibtex
@software{singh2026neatrl,
  title     = {NeatRL: Readable Single-File Reinforcement Learning Implementations in PyTorch},
  author    = {Singh, Yuvraj},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23198154},
  url       = {https://github.com/YuvrajSingh-mist/NeatRL}
}
```

## License

This project is licensed under the [Apache License 2.0](LICENSE) for code, and [CC BY 4.0](LICENSE-DATASET) for documentation, diagrams, benchmark results and published artifacts. See [NOTICE](NOTICE) and [CITATION.cff](CITATION.cff).

---

**Made with ❤️ for the RL community**
