Metadata-Version: 2.4
Name: opentau
Version: 0.1.0
Summary: OpenTau: Tensor's VLA Training Infrastructure for Real-World Robotics in Pytorch
Author-email: Shuheng Liu <wish1104@icloud.com>, William Yue <williamyue37@gmail.com>, Akshay Shah <akshayhitendrashah@gmail.com>, Xingrui Gu <xingrui_gu@berkeley.edu>
License: Apache-2.0
Project-URL: homepage, https://github.com/TensorAuto/OpenTau
Project-URL: issues, https://github.com/TensorAuto/OpenTau/issues
Project-URL: huggingface, https://huggingface.co/TensorAuto
Keywords: robotics,deep learning,pytorch
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.10
Requires-Python: ==3.10.*
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cmake>=3.29.0.1
Requires-Dist: datasets>=2.19.0
Requires-Dist: deepdiff>=7.0.1
Requires-Dist: diffusers>=0.27.2
Requires-Dist: draccus>=0.10.0
Requires-Dist: einops>=0.8.0
Requires-Dist: flask>=3.0.3
Requires-Dist: gdown>=5.1.0
Requires-Dist: h5py>=3.10.0
Requires-Dist: huggingface-hub[cli,hf-transfer]>=0.27.1; python_version < "4.0"
Requires-Dist: imageio[ffmpeg]>=2.34.0
Requires-Dist: jsonlines>=4.0.0
Requires-Dist: numba>=0.62.0
Requires-Dist: omegaconf>=2.3.0
Requires-Dist: opencv-python-headless>=4.9.0
Requires-Dist: packaging>=24.2
Requires-Dist: av>=12.0.5
Requires-Dist: pymunk>=6.6.0
Requires-Dist: pynput>=1.7.7
Requires-Dist: pyzmq>=26.2.1
Requires-Dist: rerun-sdk>=0.21.0
Requires-Dist: termcolor>=2.4.0
Requires-Dist: torch<2.8.0,>=2.7.1
Requires-Dist: torchcodec<0.5.0,>=0.4.0; sys_platform != "win32" and (sys_platform != "linux" or (platform_machine != "aarch64" and platform_machine != "arm64" and platform_machine != "armv7l")) and (sys_platform != "darwin" or platform_machine != "x86_64")
Requires-Dist: torchvision<0.23.0,>=0.22.1
Requires-Dist: wandb>=0.16.3
Requires-Dist: zarr>=2.17.0
Requires-Dist: scikit-learn>=1.7.1
Requires-Dist: onnx>=1.18.0
Requires-Dist: onnxruntime>=1.22.1; sys_platform == "darwin" or platform_machine == "arm64" or platform_machine == "aarch64"
Requires-Dist: onnxruntime-gpu>=1.22.0; (sys_platform == "linux" and platform_machine == "x86_64") or (sys_platform == "win32" and (platform_machine == "AMD64" or platform_machine == "x86_64"))
Requires-Dist: onnxscript>=0.3.1
Requires-Dist: onnx-ir>=0.1.4
Requires-Dist: opentau-transformers==4.53.3
Requires-Dist: scipy>=1.15.2
Requires-Dist: pytest>=8.1.0
Requires-Dist: pytest-cov>=5.0.0
Requires-Dist: pyserial>=3.5
Requires-Dist: pytest-xdist>=3.8.0
Requires-Dist: scikit-image>=0.23.2
Requires-Dist: pandas>=2.2.2
Requires-Dist: accelerate>=1.4.0
Requires-Dist: deepspeed>=0.17.1
Provides-Extra: dev
Requires-Dist: pre-commit>=3.7.0; extra == "dev"
Requires-Dist: debugpy>=1.8.1; extra == "dev"
Requires-Dist: pytest>=8.1.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
Requires-Dist: pyserial>=3.5; extra == "dev"
Requires-Dist: pytest-xdist>=3.8.0; extra == "dev"
Requires-Dist: sphinx>=8.1.3; extra == "dev"
Requires-Dist: sphinx-rtd-theme>=3.0.1; extra == "dev"
Requires-Dist: sphinx-copybutton>=0.5.2; extra == "dev"
Requires-Dist: sphinx-design>=0.6.1; extra == "dev"
Requires-Dist: myst-parser>=4.0.0; extra == "dev"
Provides-Extra: openai
Requires-Dist: openai>=1.0.0; extra == "openai"
Requires-Dist: python-dotenv>=1.0.0; extra == "openai"
Provides-Extra: libero
Requires-Dist: cmake<4; extra == "libero"
Requires-Dist: ninja; extra == "libero"
Requires-Dist: bddl==1.0.1; extra == "libero"
Requires-Dist: easydict==1.9; extra == "libero"
Requires-Dist: future==0.18.2; extra == "libero"
Requires-Dist: matplotlib>=3.5.3; extra == "libero"
Requires-Dist: robomimic==0.2.0; extra == "libero"
Requires-Dist: robosuite==1.4.0; extra == "libero"
Requires-Dist: thop==0.1.1.post2209072238; extra == "libero"
Requires-Dist: mujoco>=3.3.5; extra == "libero"
Requires-Dist: PyOpenGL==3.1.7; extra == "libero"
Requires-Dist: libero; extra == "libero"
Requires-Dist: numpy<2; extra == "libero"
Requires-Dist: gym<0.27,>=0.25; extra == "libero"
Requires-Dist: pyopengl-accelerate==3.1.7; sys_platform == "linux" and extra == "libero"
Requires-Dist: gymnasium[other]>=0.29; extra == "libero"
Dynamic: license-file

<p align="center">
  <a href="https://www.tensor.auto">
    <img src="assets/logo.png" alt="Logo">
  </a>
</p>

# OpenTau - Train VLA models with state-of-the-art techniques by Tensor

At Tensor, we are pushing the frontier of large foundation models for physical AI. In robot learning, a vision-language-action (VLA) model is a multimodal foundation model that integrates vision, language, and action. Today, VLA represents the leading approach for embodied AI, spanning autonomous driving, robot manipulation, and navigation.

OpenTau is Tensor’s open-source training toolchain for frontier VLA models—designed to make training reproducible, accessible, and scalable. At Tensor, we believe in open research and reproducible progress for the robotics community. By open-sourcing our training toolchain, we aim to expand knowledge sharing and accelerate scientific progress that others can reproduce.

Whether you use the official OpenPi codebase or LeRobot’s reimplementation, you may still be missing key components. OpenTau implements these key capabilities in one place:

- Co-training on an adjustable mixture of heterogeneous datasets
- Discrete actions for fast VLM convergence in $\pi_{0.5}$
- Knowledge insulation between the VLM backbone and the action expert
- Dropout in the VLM to reduce overfitting
- A reinforcement learning pipeline described in $\pi^*_{0.6}$
- And more...

OpenTau ($\tau$) is a tool developed by *[Tensor][1]* to bridge this gap, and we also use it internally to train our proprietary in-house models. Our goal is to help you train VLAs on any dataset while fully leveraging state-of-the-art techniques. We plan to continuously upgrade this repository to keep pace with the state of the art in the robotics community.

| Features                                                 | OpenPi                  | LeRobot                          | **OpenTau** |
| -------------------------------------------------------: | :---------------------: | :------------------------------: | :---------: |
| Co-training with Heterogeneous Datasets                  | ❌                       | ❌                                | ✅           |
| Discrete Actions Training in $\pi_{0.5}$                 | ❌                       | ❌                                | ✅           |
| Knowledge Insulation (KI) between VLM and Action Decoder | ❌                       | ❌                                | ✅           |
| Dropout Layers in PaliGemma                              | ✅ (Jax) <br>❌ (PyTorch) | ❌                                | ✅           |
| Multi-Node and Multi-GPU Training                        | ❌                       | ✅                                | ✅           |
| Fully Functioning $\pi_{0.5}$ Checkpoint                 | ✅                       | ❌ <br> (Missing Text Embeddings) | ✅           |
| Simulation Environments for Evaluating Models            | ❌                       | ✅                                | ✅           |
| $\pi^{*}_{0.6}$ style Reinforcement Learning Pipeline    | ❌                       | ❌                                | ✅           |
| Framework                                                | Jax / PyTorch           | PyTorch                          | PyTorch     |

## Quick Start
If you are familiar with LeRobot, getting started with OpenTau is very easy.
Because OpenTau is a fork of the popular LeRobot repository, any LeRobot-compliant policy and dataset can be used directly with OpenTau.
Check out our documentation to get started quickly.
We provide a quick start guide to help you get started with OpenTau.

For using local notebooks to train and evaluate models, find the notebooks at `notebooks/pi05_training.ipynb` and `notebooks/pi05_evaluation_only.ipynb`.

For using the Google Colab notebooks to train and evaluate models, find the colab notebooks here: [pi05_training](https://colab.research.google.com/drive/1DeU0lNnEzs1KHo0Nkgh4YKBr-xu9moBM?usp=sharing) and [pi05_evaluation_only](https://colab.research.google.com/drive/1U_AyuH9WYMT4anEWvsOtIT7g01jA0WGm?usp=sharing) respectively.

## Checkpoints
We provide fully functioning $\pi_{0.5}$ checkpoints trained with high success rates. We plan to release more models in the near future.

| Model Checkpoint              | Description                                                                                                   | Success Rate (%)                                                   |
|-------------------------------|---------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|
| [TensorAuto/tPi0.5-libero][2] | A $\pi_{0.5}$ model checkpoint trained on the LIBERO dataset with discrete actions and knowledge insulation.  | 98.4% (10) <br> 97.6% (Goal) <br> 100% (Object) <br> 98% (Spatial) |
| [TensorAuto/pi05_base][5]     | A $\pi_{0.5}$ model checkpoint converted from the official openpi checkpoint, with language embeddings added. | N/A                                                                |
| More coming soon...           |                                                                                                               |                                                                    |

## Acknowledgements

This project builds on the $\pi$ series of [papers][3] and many other open-source efforts—especially [LeRobot][4]—for re-implementing the $\pi$ models and helping standardize training infrastructure. OpenTau extends these foundations to provide a more accessible, comprehensive toolchain for training vision-language-action agents.

[1]:	https://www.tensor.ai
[2]:	https://huggingface.co/TensorAuto/tPi0.5-libero
[3]:	https://www.pi.website/blog
[4]:	https://huggingface.co/lerobot
[5]:    https://huggingface.co/TensorAuto/pi05_base
