Metadata-Version: 2.4
Name: xlab-instinct-rl
Version: 1.0.3
Summary: XLab-maintained InstinctRL with distributed checkpoint resume fixes
Home-page: https://github.com/Yangrurui/XLab
Author: Ziwen Zhuang
Author-email: 
License: CC-BY-NC-4.0 AND BSD-3-Clause
Project-URL: Upstream, https://github.com/project-instinct/instinct_rl
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: licenses/dependencies/numpy_license.txt
License-File: licenses/dependencies/torch_license.txt
Requires-Dist: torch>=2.4.0
Requires-Dist: torchvision>=0.5.0
Requires-Dist: numpy>=1.16.4
Requires-Dist: tensorboardX
Requires-Dist: tensorboard
Requires-Dist: tabulate
Requires-Dist: GitPython
Requires-Dist: onnx
Requires-Dist: onnxscript
Dynamic: author
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# XLab-maintained Instinct RL

This distribution is named `xlab-instinct-rl`; Python imports remain
`instinct_rl`. It is maintained for XLab, based on upstream
`project-instinct/instinct_rl` commit `3a2844890387eda6d93a4465cdef9e767aba8546`.
Release 1.0.3 includes XLab repair `097212a`: every distributed worker restores
its checkpoint to its local device, including optimizer and adaptive learning
rate state. No other algorithm changes are included.

After the release is available on PyPI:

```bash
python -m pip uninstall instinct-rl
python -m pip install xlab-instinct-rl==1.0.3
```

The two distribution names share the same import package and must not be
installed together. XLab validates Python 3.11; the declared minimum is Python
3.10, required by the Torch dependency. Existing training processes retain
already imported code.

Original authorship, license notices, and upstream documentation follow.


## Warning
This codebase is under [CC BY-NC 4.0 license](LICENSE), with inherited license in IsaacLab. You may not use the material for commercial purposes, e.g., to make demos to advertise your commercial products or wrap the code for your own commercial purposes.

## Contributing
See our [Contributor Agreement](CONTRIBUTOR_AGREEMENT.md) for contribution guidelines. By contributing or submitting a pull request, you agree to transfer copyright ownership of your contributions to the project maintainers.

See [CONTRIBUTORS.md](CONTRIBUTORS.md) for a list of acknowledged contributors.

---

## Install

- Clone this repository separately from the Project Instinct installation:

  ```bash
  # Option 1: HTTPS
  git clone https://github.com/project-instinct/instinct_rl.git
  # Option 2: SSH
  git clone git@github.com:project-instinct/instinct_rl.git
  ```

- Install using any python interpreter

  ```bash
  python -m pip install -e instinct_rl
  ```

---

## Important New Concepts

### General Observation format

- **obs_format**: An OrderedDict specifying the format of input observations.
    key: obs_segment name, specifically for algorithm requirements, as obs_group in ManagerBasedRlEnv in IsaacLab
    value: an obs_segment object (see below)

- **obs_segment**: An OrderedDict specifying the segment of the observation.
    key: obs term name, as obs_term_name in ManagerBasedRlEnv in IsaacLab
    value: the shape of each part of the observation.

- **obs_pack**: The dictionary of observations following the definition of obs_format.
    key: obs_segment name as in observation_manager in ManagerBasedRlEnv in IsaacLab
    value: A flattened tensor/vector of each observation (required by the algorithm).

- **obs_component**: As obs_term in ManagerBasedRlEnv in IsaacLab.

---

## How to use the swappable algorithms and network modules design

In `on_policy_runner.py`, we have the following code:

```python
actor_critic = modules.build_actor_critic(
    self.policy_cfg.pop("class_name"),
    self.policy_cfg,
    obs_format,
    num_actions=env.num_actions,
    num_rewards=env.num_rewards,
).to(self.device)
```

This code is used to build the actor critic network. The `class_name` is the name of the actor critic class, and the `policy_cfg` is the configuration of the actor critic. The `obs_format` is the observation format, and the `num_actions` is the number of actions, and the `num_rewards` is the number of rewards.

The `modules.build_actor_critic` function is used to build the actor critic network. It is a factory function that builds the actor critic network based on the `class_name` and `policy_cfg`.

The `class_name` is the name of the actor critic class, which can be a class implemented in this repository, or a full import path to your own actor critic class in `module_name:class_name` pattern.

---

## Algorithms

### PPO

Standard Proximal Policy Optimization (PPO) algorithm for reinforcement learning. Implements the PPO objective with clipped surrogate loss, value function loss, and entropy regularization. Supports multiple learning epochs and mini-batch updates per environment step.

### State Estimator

Algorithm for learning state representations and estimation from partial observations. Combines reconstruction and prediction objectives to learn meaningful latent state representations that can be used for control.

### AMP (WASABI)

Implementation of WASABI, starting from the algorithm framework of AMP. Uses adversarial training with a discriminator to learn motion priors from expert demonstrations. Supports multiple discriminator architectures, including BCE loss, Wasserstein loss, and MSELoss. Includes gradient penalty and various discriminator architectures.

### Distillation and DAgger (TPPO)

Teacher-student distillation framework that extends PPO with teacher network guidance. Implements Dataset Aggregation (DAgger) and knowledge distillation techniques. Allows learning from expert demonstrations while maintaining PPO's stability guarantees. Supports various teacher action selection probabilities and distillation loss coefficients.

### VAE Distillation

Teacher-student distillation framework that extends TPPO with a VAE student network. Uses a VAE to encode and decode to generate the student's action. It generates additional latent distribution separate from the action distribution, which enables the potential of VAE-distillation combined with PPO.

---

## Network modules

### Actor Critic

Standard actor-critic architecture with separate policy (actor) and value (critic) networks. Supports configurable hidden dimensions, activation functions, and observation processing. Implements both continuous and discrete action spaces with proper probability distributions.

### MoE Actor Critic

Mixture of Experts (MoE) extension of the actor-critic architecture. Uses multiple expert networks with a gating mechanism to specialize different experts on different parts of the state/action space. Can improve performance on complex tasks by allowing specialization while maintaining parameter efficiency.

### VAE Actor Critic

Variational Autoencoder (VAE) extension of the actor-critic architecture. Uses a VAE to learn a latent representation of the observation, and then uses the actor to predict the action from the latent representation. Can be used for dimensionality reduction, feature learning, or as part of a larger reinforcement learning pipeline.

### Encoder related modules

Collection of encoder architectures for processing different input modalities:

- **Encoder Actor Critic**: Actor-critic networks with encoder backbones for processing high-dimensional inputs like images or complex state representations
- **All Mixer**: Multi-modal input processing that combines different observation types
- **VQ-VAE**: Vector Quantized Variational Autoencoder for discrete latent representations
- **State Estimator**: Networks for estimating hidden states from partial observations
