Metadata-Version: 2.5
Name: vlastudio
Version: 0.2.0
Summary: Extensible robotics policy training and deployment
Project-URL: Homepage, https://github.com/WwZzz/VLAStudio
Project-URL: Documentation, https://github.com/WwZzz/VLAStudio#readme
Project-URL: Repository, https://github.com/WwZzz/VLAStudio
Project-URL: Issues, https://github.com/WwZzz/VLAStudio/issues
Author: VLAStudio contributors
License: You are free to:
        Share — copy and redistribute the material in any medium or format
        Adapt — remix, transform, and build upon the material
        The licensor cannot revoke these freedoms as long as you follow the license terms.
        Under the following terms:
        Attribution — You must give appropriate credit , provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
        NonCommercial — You may not use the material for commercial purposes .
        No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
        Notices:
        You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation .
        
        No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
License-File: LICENSE
Keywords: imitation-learning,robotics,vision-language-action
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <3.13,>=3.10
Requires-Dist: accelerate<=1.12.0,>=1.0.1
Requires-Dist: av<16,>=12
Requires-Dist: diffusers<=0.35.1,>=0.30.3
Requires-Dist: dm-control<=1.0.34,>=1.0.21
Requires-Dist: editables<1,>=0.3
Requires-Dist: einops==0.8.1
Requires-Dist: filelock<4,>=3.16
Requires-Dist: h5py==3.12.1
Requires-Dist: huggingface-hub<1,>=0.29
Requires-Dist: imageio-ffmpeg==0.6.0
Requires-Dist: imageio==2.37.0
Requires-Dist: loguru<1,>=0.7
Requires-Dist: matplotlib==3.9.4
Requires-Dist: mujoco<=3.3.6,>=3.2.0
Requires-Dist: numpy==1.26.4
Requires-Dist: opencv-python<=4.10.0.84,>=4.9
Requires-Dist: packaging>=24
Requires-Dist: pillow<13,>=11.3
Requires-Dist: platformdirs<5,>=4
Requires-Dist: protobuf==5.29.6
Requires-Dist: psutil==6.1.1
Requires-Dist: pyarrow>=14
Requires-Dist: pyyaml<7,>=6
Requires-Dist: requests==2.32.5
Requires-Dist: scipy==1.14.1
Requires-Dist: tensorboardx==2.6.4
Requires-Dist: timm<=1.0.15,>=0.9.16
Requires-Dist: tomli>=2; python_version < '3.11'
Requires-Dist: torch<2.8,>=2.4
Requires-Dist: torchdata==0.9.0
Requires-Dist: torchvision<0.23,>=0.19
Requires-Dist: tqdm<5,>=4.67
Requires-Dist: transformers<=4.53.2,>=4.40.1
Requires-Dist: websockets==13.1
Provides-Extra: act
Provides-Extra: aloha
Provides-Extra: dev
Requires-Dist: build>=1; extra == 'dev'
Requires-Dist: hatchling; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Provides-Extra: remote-eval
Provides-Extra: serve-http
Requires-Dist: fastapi==0.115.6; extra == 'serve-http'
Requires-Dist: uvicorn==0.34.0; extra == 'serve-http'
Description-Content-Type: text/markdown

<div align="center">
  <img src='https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/vlastudio.png'  width="200"/>
<h1> VLAStudio: A Modular Learning Playground for Robotics
</h1>


</div>


<table>
<caption><em>demos generated by policies trained in VLAStudio</em></caption>
<tr>
<td align="center">ACT - RLBench</td>
<td align="center">Diffusion Policy - ALOHA-Sim</td>
<td align="center">ACT - Square</td>
<td align="center">FlowMatching - Transport</td>
<td align="center">SmolVLA - MetaWorld</td>
</tr>
<tr>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/act_rlbench.gif" height="200"></td>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/diffusion_aloha_transfer.gif" height="200"></td>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/act_robomimic_square.gif" height="200"></td>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/flowmatching_transport.gif" height="200"></td>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/fig_smolvla_mw41.gif" height="200"></td>
</tr>
<tr>
<td align="center">Koch - Pick & Place -Inference</td>
<td align="center">SO101 - Fold Tower -Inference</td>
<td align="center">BC_MLP - AdroitHandDoor</td>
<td align="center">Pi0 - LIBERO</td>
<td align="center">ACT - RoboTwin</td>
</tr>
<tr>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/fig_koch.gif" height="200"></td>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/so101_fold.gif" height="200"></td>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/fig_door.gif" height="200"></td>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/pi0_libero.gif" height="200"></td>
<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/robotwin_act.gif" height="200"></td>
</tr>
</table>


VLAStudio is a Python library for robot learning. Write your own `run.py` to load demonstrations, train imitation-learning and vision-language-action policies, evaluate in simulation, and deploy on real robots. Compose these steps with ordinary Python functions and run your file with `python run.py`.

- **Train and fine-tune** ACT, Diffusion Policy, SmolVLA, Pi0/Pi0.5, and other policies with a shared Python API.
- **Combine data sources** through HDF5, LeRobot, and RLDS loaders and shared data processing.
- **Evaluate and deploy** locally or through a policy server, with simulation benchmarks and real robot integrations.
- **Extend the library** with your own policy, dataset, simulator, or robot without rewriting the pipeline.
- **Choose your own Python environment.** Install ACT, Diffusion Policy and ALOHA-Sim with pip; use uv virtual environments and requirements files for other models.

## 🛠️ Installation

### Install the default environment

With Python 3.10–3.12 in an activated environment:

```bash
python -m pip install vlastudio
```

For development or to run the repository examples:

Clone the main branch, then install with pip in a uv virtual environment:

```bash
git clone https://github.com/WwZzz/VLAStudio.git
cd VLAStudio
python -m pip install uv
python -m uv venv --python 3.10 --seed .venv
source .venv/bin/activate
python -m pip install -e .
```

That is the complete Python dependency installation for **ACT, Diffusion Policy,
shared HDF5/LeRobot parquet and video processing, and ALOHA-Sim**. No additional
model or simulator package installation is needed for the ACT/DP ALOHA examples.
Provide demonstrations before training; the policies may download pretrained
ResNet weights on first use. GPU execution needs a compatible NVIDIA driver,
and headless rendering needs the host's OpenGL/EGL libraries.

On Windows, activate with `.venv\Scripts\Activate.ps1`. Use `python -m pip install .`
for a non-editable installation. Without a checkout, install the main branch:

```bash
python -m pip install "git+https://github.com/WwZzz/VLAStudio.git@main"
```

`import vlastudio` itself loads only Python's standard library. It never imports
Torch, starts a simulator, creates an environment, or installs packages.

### Other models: a separate environment and a requirements file

For example, install SmolVLA and LIBERO in a separate environment:

```bash
deactivate
python -m uv venv --python 3.10 --seed .venv-smolvla
source .venv-smolvla/bin/activate
python -m pip install -e . -r requirements/smolvla.txt -r requirements/libero.txt
```

Install the project and selected requirements in the **same pip invocation** so
the resolver considers their versions together. The model files share a base
and select model-specific versions. Simulator assets are prepared separately;
follow the corresponding example before evaluation.

The checkout also supports `python -m uv sync --python 3.10` after installing uv
in the environment; its lock includes the default ACT/DP/ALOHA packages. Use
ordinary `python run.py` after activation. For optional models, follow their
requirements-file instructions instead of syncing the default lock.

`load_policy`, `load_dataset`, and `load_env` report missing or incompatible
dependencies and print an external pip installation command. For a read-only check:

```python
import vlastudio as vla

plan = vla.environment_plan(policy="act", environment="aloha_sim")
print(plan["issues"])
print(plan["commands"]["install_current"])
```

See the [environment guide](docs/environments.md) for other models, Python versions,
CUDA wheels, and custom dependency metadata. Installation is performed only by
your terminal's pip/uv commands; workflow code uses the installed environment.

## 🚀 Write Your Own run.py

### ACT on ALOHA-Sim

Prepare the demonstrations at `data/sim_transfer_cube_scripted`, or use a task YAML with your own dataset path. The built-in starting point is [sim_transfer_cube_scripted.yaml](src/vlastudio/configs/task/sim_transfer_cube_scripted.yaml).

Save this as `run.py` in your project:

```python
import vlastudio as vla


def main():
    dataset = vla.load_dataset("sim_transfer_cube_scripted")
    policy = vla.load_policy("act")
    result = vla.train(
        policy, dataset, "default", output_dir="checkpoints/act_aloha"
    )
    benchmark = vla.load_env("aloha_transfer")
    evaluation = benchmark.evaluate(result.policy, output_dir="results/act_aloha")
    print(evaluation.metrics)


if __name__ == "__main__":
    main()
```

```bash
python run.py
```

The script uses your current Python environment. Paths resolve relative to your working directory; pass your own YAML paths or built-in names to the loaders. You can freely add preprocessing, logging, evaluation logic, or other library calls to this file.

The [ACT example](examples/01_train_and_eval_act_on_aloha) provides the same workflow:

```bash
python examples/01_train_and_eval_act_on_aloha/train_and_evaluate.py
```

### Diffusion Policy

The default installation above already includes Diffusion Policy and ALOHA-Sim. Write `run_dp.py`:

```python
import vlastudio as vla


def main():
    dataset = vla.load_dataset("sim_transfer_cube_scripted")
    policy = vla.load_policy("diffusion_policy")
    vla.train(
        policy, dataset, "dp", output_dir="checkpoints/dp_aloha",
        overrides={"training.max_steps": 200000, "training.save_steps": 10000},
    )
    vla.load_env("aloha_transfer").evaluate(policy, output_dir="results/dp_aloha")


if __name__ == "__main__":
    main()
```

```bash
python run_dp.py
```

### SmolVLA on MetaWorld

In a separate Python 3.10 environment, run `python -m pip install -e . -r requirements/smolvla.txt`. The built-in task uses the LeRobot MetaWorld dataset. Provide access to that dataset and the pretrained policy weights.

Write `train_smolvla.py`:

```python
import vlastudio as vla


def main():
    dataset = vla.load_dataset("metaworld")
    policy = vla.load_policy("smolvla")
    vla.train(
        policy, dataset, "smolvla", output_dir="checkpoints/smolvla_metaworld",
        overrides={"training.max_steps": 100000,
                   "training.per_device_train_batch_size": 64},
    )


if __name__ == "__main__":
    main()
```

```bash
python train_smolvla.py
```

Use the policy-server workflow below to evaluate with a simulator in another environment.

### Runnable examples

Each example shows its own environment setup and Python files. Run from the repository root after installing the required dependencies.

| Goal | Example |
| --- | --- |
| Train and evaluate ACT | [01 · ACT on ALOHA](examples/01_train_and_eval_act_on_aloha) |
| Serve a policy and evaluate remotely | [02 · Remote inference](examples/02_remote_inference) |
| Write a custom policy | [03 · Custom policy](examples/03_customize_policy) |
| Train and evaluate across tasks | [04 · Multitask policy](examples/04_multitask_policy) |
| Fine-tune Pi0.5 | [05 · Pi0.5](examples/05_finetune_vla_pi05) |
| Write and control a custom robot | [06 · Custom robot](examples/06_custom_robot) |
| Integrate a custom simulator | [07 · LIBERO Plus](examples/07_custom_sim_env_libero_plus) |
| Train and evaluate FastWAM | [08 · FastWAM on LIBERO](examples/08_fastwam_libero) |

## Overview

![framework](https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/fig_1_ilstudio.png)

## 🌐 Policy Server

Write `serve.py` in the model environment:

```python
import vlastudio as vla


if __name__ == "__main__":
    policy = vla.load_policy("smolvla", checkpoint="checkpoints/smolvla_metaworld")
    vla.serve(policy, address="127.0.0.1:5000")
```

In a separate simulation environment, run `python -m pip install -e . -r requirements/metaworld.txt`, then write `evaluate.py`:

```python
import vlastudio as vla


if __name__ == "__main__":
    policy = vla.connect_policy("127.0.0.1:5000")
    benchmark = vla.load_env("metaworld.easy")
    result = benchmark.evaluate(
        policy, output_dir="results/smolvla_metaworld", num_rollout=10, batch_size=10
    )
    print(result.metrics)
```

Start `python serve.py` in the model environment, wait for the server to listen, then start `python evaluate.py` in the simulation environment. The server handles inference while the client handles observations and actions. See [Example 02](examples/02_remote_inference) for runnable files.

## 🤖 Real Robots and Data Collection

For SO101, create a dedicated Python 3.10 environment and install the model and motor SDK together:

```bash
python -m pip install -e . -r requirements/smolvla.txt "lerobot[feetech]==0.3.3"
```

Customize serial ports, camera devices, IDs, and calibration paths in your robot and teleoperator YAMLs. Starting points: [so101.yaml](src/vlastudio/configs/robot/so101.yaml), [so101_leader.yaml](src/vlastudio/configs/teleop/so101_leader.yaml).

Write `collect.py`:

```python
import vlastudio as vla


if __name__ == "__main__":
    vla.collect(
        "so101", teleop="so101_leader", output_dir="data/so101_pick_place",
        dataset_format="hdf5", task="pick up the cube",
    )
```

Write `infer.py` to use an already running policy server:

```python
import vlastudio as vla


if __name__ == "__main__":
    policy = vla.connect_policy("127.0.0.1:5000")
    vla.infer("so101", policy)
```

Run `python collect.py` to collect demonstrations or `python infer.py` to deploy. Pass custom YAML paths instead of built-in names for your own robot. See the [SO101 guide](src/vlastudio/deploy/robot/so101/README.md).

## 🎨 Policy Gallery

| **Policy**            | **Reference**                                                                                          |
| --------------------- | ------------------------------------------------------------------------------------------------------ |
| ACT                   | [[1]](https://arxiv.org/abs/2304.13705)                                                                |
| Diffusion Policy      | [[2]](https://arxiv.org/abs/2303.04137)                                                                |
| Qwen2VL+DP            | [[3]](https://arxiv.org/abs/2308.12966) + [[2]](https://arxiv.org/abs/2303.04137)                          |
| Qwen2.5VL+DP          | [[3]](https://arxiv.org/abs/2308.12966) + [[2]](https://arxiv.org/abs/2303.04137)                          |
| DiVLA                 | [[4]](https://arxiv.org/html/2412.03293v1)                                                                |
| OpenVLA               | [[5]](https://arxiv.org/abs/2406.09246)                                                      |
| OpenVLA-OFT               | [[6]](https://arxiv.org/abs/2502.19645)                                                      |
| Pi0                   | [[7]](https://arxiv.org/html/2410.24164v1)                                                             |
| Pi0.5                   | [[8]](https://arxiv.org/abs/2504.16054)                                                             |
| MLP                   | -                                                                                                      |
| ResNet                | [[9]](https://arxiv.org/abs/1512.03385)                                                                |
| SmolVLA               | [[10]](https://arxiv.org/abs/2506.01844)                                                                |
| Octo                  | [[11]](https://octo-models.github.io/)    |
| FlowMatching                  |  -    |
| GR00T| [[12]](https://arxiv.org/abs/2503.14734)|
| X-VLA                 | [LeRobot Docs](https://huggingface.co/docs/lerobot/main/en/xvla) |

## 🏆 Simulation Benchmark Gallery

| Environment | Parallel rollouts | Need Vulkan |
|------------|--------|------------|
| aloha_sim | ✅ | ❌ |
| gymnasium_robotics | ✅ | ❌ |
| libero | ✅ | ❌ |
| metaworld | ✅ | ❌ |
| pandagym | ✅ | ❌ |
| robomimic | ✅ | ❌ |
| simplerenv | ✅ | ✅ |
| robotwin | ❌ | ✅ |
| rlbench | ❌ | ❌ |
| calvin (Under Testing) | ❌ | ❌ |
| bigym | ✅ | ❌ |
| behavior1k | ❌ | ❌ |

## 🦾 Robot Gallery
- SO101
- BimanualSO101
- Kochv1.1
- AgilexAloha (CobotMagic)


## 📊 Datasets and Extensions

VLAStudio provides HDF5, LeRobot, and RLDS loaders. Data processing and batch collation adapt each source to the selected policy. Your `run.py` can select built-in names or your own task and policy YAML files:

```python
import vlastudio as vla

dataset = vla.load_dataset("path/to/task.yaml")
policy = vla.load_policy("path/to/policy.yaml")
```

Built-in configurations live in [src/vlastudio/configs](src/vlastudio/configs). User configurations can live anywhere.

Start from the examples above to write your own components. A policy supplies `load_model` and a model with `select_action`; optional data processors, collators, and trainers customize training. The [developer guide](docs/package_runtime.md) describes package and runtime internals.

## 🩹 Troubleshooting

- **Missing or conflicting dependencies:** inspect `vla.environment_plan(...)` and use its pip command in the intended environment, or install the corresponding requirements file.
- **Headless MuJoCo on Linux:** set `os.environ["MUJOCO_GL"] = "egl"` near the start of your Python file. CPU rendering needs system OSMesa libraries and the `osmesa` backend; a desktop display can use `glfw`.
- **Old CMake projects fail with “Compatibility with CMake < 3.5 has been removed”:** set `CMAKE_POLICY_VERSION_MINIMUM=3.5` in the terminal before running pip for that optional simulator.
- **Model or simulator resources are missing:** Python dependencies, pretrained weights, datasets, and native simulator assets are separate setup steps. Follow the relevant example's README.

## 📫 Contact

- Wang Z. <zwang@stu.xmu.edu.cn>
- Yuxin G. <yuxinxmu@163.com>

## 🙏 Acknowledgements
This repo is built on the open source codebases below. Thanks to the authors' wonderful contributions.

- [Lerobot](https://github.com/huggingface/lerobot/tree/main)

- [ACT](https://github.com/MarkFzp/act-plus-plus/)

- [Octo-pytorch](https://github.com/emb-ai/octo-pytorch/)

- [Openpi](https://github.com/Physical-Intelligence/openpi)

- [OpenVLA](https://github.com/openvla/openvla/)

- [DiVLA](https://diffusion-vla.github.io/)

- [RoboTwin](https://robotwin-platform.github.io)

- [RLBench](https://github.com/stepjam/RLBench/)

- [StarVLA](https://github.com/starVLA/starVLA/)
