Metadata-Version: 2.5
Name: libero-pro-benchmark
Version: 0.1.1
Import-Name: libero
Import-Name: libero_pro_benchmark
Import-Name: perturbation
Summary: Standalone NumPy-based LIBERO and LIBERO-PRO simulation benchmark
License-Expression: MIT
License-File: LICENSE
Requires-Python: <3.15,>=3.10
Requires-Dist: bddl==1.0.1
Requires-Dist: cloudpickle<4,>=3
Requires-Dist: easydict<2,>=1.13
Requires-Dist: future<2,>=1.0
Requires-Dist: gymnasium<2,>=1.0
Requires-Dist: libero-pro-benchmark-assets-1==0.1.0
Requires-Dist: libero-pro-benchmark-assets-2==0.1.0
Requires-Dist: libero-pro-benchmark-assets-3==0.1.0
Requires-Dist: libero-pro-benchmark-assets-4==0.1.0
Requires-Dist: libero-pro-benchmark-assets-5==0.1.0
Requires-Dist: matplotlib<4,>=3.8
Requires-Dist: mujoco<4,>=2.3.7
Requires-Dist: numpy<3,>=1.26
Requires-Dist: opencv-python<5,>=4.8
Requires-Dist: pyyaml<7,>=6
Requires-Dist: robosuite==1.4.0
Requires-Dist: termcolor<4,>=2.4
Requires-Dist: tqdm<5,>=4.66
Provides-Extra: compatible
Requires-Dist: mujoco==2.3.7; extra == 'compatible'
Requires-Dist: numpy==1.26.4; extra == 'compatible'
Requires-Dist: opencv-python<4.12; extra == 'compatible'
Provides-Extra: datasets
Requires-Dist: h5py<4,>=3.11; extra == 'datasets'
Provides-Extra: video
Requires-Dist: imageio-ffmpeg<1,>=0.5; extra == 'video'
Requires-Dist: imageio<3,>=2.34; extra == 'video'
Description-Content-Type: text/markdown

# LIBERO-PRO-Benchmark

A PyTorch-free simulation benchmark for evaluating VLA policies on LIBERO and LIBERO-PRO, with the original benchmark imports and bundled evaluation resources.

## What this package provides

`libero-pro-benchmark` is an independent, modernized extraction of the benchmark from [LIBERO-PRO](https://github.com/Zxy-MLlab/LIBERO-PRO), which builds on [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO). It provides task definitions, MuJoCo environments, success checks, initial states, and the upstream perturbation utilities. It is not an official upstream release.

**One installation supports both LIBERO and LIBERO-PRO evaluation.** You do not need to install the original repositories alongside it. Existing evaluation code can keep imports such as `from libero.libero import benchmark` and `from libero.libero.envs import OffScreenRenderEnv`.

The benchmark does not install PyTorch, a VLA model, or a training framework. Supply your own policy and its inference dependencies. There is no CPU, CUDA, macOS, or Linux extra to select: the policy's device is controlled by the calling application.

Model assets, task descriptions, and evaluation initial states are installed with the package. This includes the official LIBERO-PRO evaluation resources that upstream distributes separately; no tasks have been invented or added to the upstream registry. No manual resource copying or first-use download is needed for the covered tasks.

## Why this package?

A VLA evaluator needs the simulation environment, task instructions, initial states, and success checks. The upstream LIBERO-PRO repository also includes policy training code and setup instructions tied to older Python and PyTorch releases. Recreating that full stack can add unnecessary dependencies and make integration with a policy's own environment harder.

This package makes the benchmark usable as a normal dependency in an existing evaluation project:

- **Simpler installation.** Install with `pip install libero-pro-benchmark` or `uv add libero-pro-benchmark`. Model assets and the official separately distributed evaluation resources are installed automatically for the covered tasks.
- **Independent policy dependencies.** The benchmark uses NumPy initial states and does not require PyTorch. Your policy can choose its own inference framework and device without inheriting the upstream training stack.
- **Modern simulation setup.** Use supported current Python and MuJoCo versions within the declared constraints, or select `compatible` for the measured reference dependency stack. Simulator upgrades can change trajectories and rendering.
- **Less evaluation-code migration.** Retain the original `libero.libero` imports and benchmark APIs. Task definitions and ordering are preserved, and initial-state conversion preserves the original numeric bytes.
- **LIBERO and LIBERO-PRO together.** One benchmark dependency provides the five standard LIBERO suites and the covered PRO extensions, with explicit resource-availability checks for other upstream entries.

It is intended for researchers who already have a policy or evaluation runner and need reusable benchmark environments and tasks. The contribution is installation, dependency separation, and resource packaging; it does not introduce a new benchmark protocol, new tasks, or a claim of identical physics across simulator versions.

## Install

Use a virtual environment with CPython 3.10–3.14. The project targets macOS and Linux. Native installation and rendering have been verified on macOS / Apple Silicon; Linux still awaits real-machine validation.

With pip, in your active environment:

```bash
python -m pip install libero-pro-benchmark
```

Or add it to your own uv project:

```bash
uv add libero-pro-benchmark
```

The main package automatically installs five required `libero-pro-benchmark-assets-*` packages. You do not install these individually or choose a subset. The model resource wheels total approximately 340 MiB, in addition to the benchmark and simulation dependencies. After installation, evaluation resources are available offline.

Use a clean environment when replacing upstream LIBERO or LIBERO-PRO. Remove the upstream distribution from your project's dependencies before adding this package; they share the `libero` import namespace and must not be used together. Your policy may still use PyTorch in the same environment.

### Reference simulation dependencies (`compatible`)

The default installation allows newer MuJoCo and NumPy versions within the package constraints. For the measured reference stack, install the `compatible` extra in a **Python 3.10 or 3.11 environment**; Python 3.11 is recommended:

```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install "libero-pro-benchmark[compatible]"
```

For a uv project targeting Python 3.10/3.11:

```bash
uv add "libero-pro-benchmark[compatible]"
```

Or install directly into a uv-managed Python 3.11 virtual environment:

```bash
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python "libero-pro-benchmark[compatible]"
```

Quote package names containing brackets when using shells such as zsh. The extra pins MuJoCo 2.3.7, NumPy 1.26.4, and OpenCV below 4.12. **It does not change your Python interpreter.** MuJoCo 2.3.7 has no Python 3.12+ wheels; use a separate Python 3.11 environment rather than trying to install this stack into Python 3.14. For uv projects, keep `requires-python` consistent with the chosen interpreter range, for example `">=3.11,<3.12"` for a Python 3.11 evaluation project.

Initial-state arrays preserve the original dtype, shape, ordering, and numeric bytes in either installation. The default simulator can produce different trajectories and images from the old stack. Reference-stack equality was measured on specific same-host trajectory and rendering samples; `compatible` is not a guarantee of identical results on every machine or every rollout. Record your Python and simulation dependency versions when reporting evaluation results.

## Which tasks are included?

### Standard LIBERO

All tasks in the five standard suites are included, with their upstream names and base ordering:

| Suite name | Task entries | Evaluation focus |
|---|---:|---|
| `libero_spatial` | 10 | Spatial relations |
| `libero_object` | 10 | Object variation |
| `libero_goal` | 10 | Goal variation |
| `libero_90` | 90 | The LIBERO-90 task collection |
| `libero_10` | 10 | The LIBERO-10 task collection |

These are 130 suite entries; suite counts are not a claim of 130 mutually distinct underlying tasks.

### LIBERO-PRO

The four base suites `libero_spatial`, `libero_object`, `libero_goal`, and `libero_10` each have the following resource-complete PRO variants:

| Suffix | Perturbation | Example suite | Entries per variant suite |
|---|---|---|---:|
| `_lan` | Language instruction | `libero_spatial_lan` | 10 |
| `_object` | Object replacement | `libero_spatial_object` | 10 |
| `_swap` | Object-position swaps | `libero_spatial_swap` | 10 |
| `_task` | Task replacement | `libero_spatial_task` | 10 |

This gives 16 PRO suites and 160 entries in addition to the standard suites. To evaluate one, select its exact suite name using the same API shown below. The perturbation utilities also retain upstream environment, object, language, swap, and task transformations; generating a custom task does not make it an official evaluation task or supply official initial states for it.

### Other registered upstream extensions

The complete upstream registry is retained: 83 suites and 960 task entries. **Registration does not mean all required resources exist.** The bundled inventory identifies 33 suites with complete known resources and 409 entries with the known required task, initialization, and model resources. The first task of each of the 33 complete suites passed reset, state restoration, and stepping; this does not certify every rollout or a policy's success rate.

Besides the standard and 16 PRO suites above, the complete suites are `libero_mine`, `libero_study_table`, `libero_object_two_normal_episodes`; the `_with_mug` variants of `libero_spatial`, `libero_object`, `libero_goal`, and `libero_10`; the `_with_yellow_book` variants of `libero_spatial`, `libero_object`, and `libero_goal`; and the `_with_milk` variants of `libero_spatial` and `libero_goal`.

Other upstream extensions have missing or empty initial states, missing task descriptions, or unavailable models. The missing resources were supplied by neither the recorded upstream checkout nor its official evaluation dataset, and are not fabricated here. Do not treat every registered extension, including the `_env` family, as runnable.

Check the installed resource coverage before choosing an extension:

```bash
libero-pro list
libero-pro audit --output resource-availability.json
```

In a uv project, prefix these commands with `uv run`. The list reports entries with required resources for each suite; the audit identifies missing resources for individual tasks. These commands do not run your policy. `audit --strict` intentionally fails when any registered entry lacks resources.

## Run an environment

This executable example restores the first official initial state and performs a short zero-action rollout. The zero actions demonstrate the API; they are not a task-solving policy or an evaluation protocol.

```python
from pathlib import Path

import numpy as np
from libero.libero import benchmark, get_libero_path
from libero.libero.envs import OffScreenRenderEnv

suite_name = "libero_spatial"  # e.g. "libero_spatial_lan" for a PRO suite
suite = benchmark.get_benchmark(suite_name)(task_order_index=0)
task_id = 0
task = suite.get_task(task_id)
initial_states = suite.get_task_init_states(task_id)
bddl_file = (
    Path(get_libero_path("bddl_files")) / task.problem_folder / task.bddl_file
)

env = OffScreenRenderEnv(
    bddl_file_name=str(bddl_file),
    camera_heights=128,
    camera_widths=128,
)
try:
    env.seed(123)
    env.reset()
    obs = env.set_init_state(initial_states[0])
    print("Instruction:", task.language)
    for step in range(20):
        # Replace with your policy's action using obs and task.language.
        action = np.zeros(env.env.action_dim)
        obs, reward, done, info = env.step(action)
        success = bool(env.check_success())
        if success or done:
            break
    print("Success:", success)
finally:
    env.close()
```

The standard Panda controller uses a seven-element action. Observations include `agentview_image`, `robot0_eye_in_hand_image`, and robot state fields. `task.language` is the task instruction to pass to a language-conditioned policy. Keep the observation preprocessing, action representation, task order, initial-state selection, and rollout horizon expected by your chosen VLA evaluation protocol. Task success is available through `env.check_success()`; do not assume every episode ending means success.

The original four-value `step()` return is preserved: `(obs, reward, done, info)`. It is not Gymnasium's five-value return format. Vector environment and segmentation imports are also retained under `libero.libero.envs`.

The alternate imports `from libero_pro_benchmark import benchmark, get_libero_path` and `from libero_pro_benchmark.envs import OffScreenRenderEnv` expose the same implementation objects. The installation name is `libero-pro-benchmark`; the original namespace remains `libero.libero`.

For a quick installation check, without a policy:

```bash
libero-pro smoke --suite libero_spatial --steps 2
libero-pro smoke --suite libero_spatial_lan --steps 2
```

Rendering needs a working graphics backend. On headless Linux with EGL drivers installed, set `MUJOCO_GL=egl` before starting Python. macOS uses its native backend; do not copy the Linux EGL setting into a macOS environment. `libero-pro smoke --no-render` checks state simulation without camera rendering.

## Use it with an existing VLA evaluator

Replace the upstream benchmark dependency with `libero-pro-benchmark`, retain the original imports, and provide your evaluator's own model/inference dependencies. Code using the public task and environment APIs can continue to use them without changing those imports.

There are explicit migration differences for code bypassing the APIs:

- Initial states are stored as losslessly converted NPZ arrays. Use `suite.get_task_init_states(task_id)`; it returns a NumPy array. Code calling `torch.load()` on an initialization file directly must change its loading call.
- Use `get_libero_path("assets")`, `get_libero_path("bddl_files")`, and `get_libero_path("init_states")` for installed resources. Paths constructed relative to an upstream checkout or a module's `__file__` are not portable to this installation.
- Upstream policy training code and its training loaders are not included. This package supplies the benchmark environment for your evaluator.
- The old implicit `~/.libero` configuration is not read. For explicit resource overrides, set `LIBERO_CONFIG_PATH` to a directory containing a `config.yaml`. An initial-state override must contain NPZ-format states.

The package does not implement a particular model service, evaluation leaderboard, or universal VLA adapter. Your evaluator remains responsible for inference, episode scheduling, metrics, and its benchmark protocol.

## Optional demonstrations and video tools

Neither extra is needed for ordinary online evaluation:

| Extra | What it enables | Installation |
|---|---|---|
| `datasets` | h5py and the upstream `get_dataset_info` HDF5 inspection helper | `python -m pip install "libero-pro-benchmark[datasets]"` |
| `video` | The upstream `VideoWriter` with imageio and FFmpeg | `python -m pip install "libero-pro-benchmark[video]"` |

Extras can be combined, for example `"libero-pro-benchmark[compatible,video]"` in Python 3.11. In uv projects, use the same requirement with `uv add`.

The [official LIBERO demonstration dataset](https://huggingface.co/datasets/yifengzhu-hf/LIBERO-datasets) contains HDF5 demonstration trajectories for training or analysis. Installing `datasets` does not download it and does not add a training framework. The [official LIBERO-PRO evaluation dataset](https://huggingface.co/datasets/zhouxueyang/LIBERO-Pro) supplies task descriptions and initial states, rather than HDF5 demonstrations; the evaluation resources covered here are already bundled.

Installing `video` does not record automatically. Callers explicitly create `libero.libero.utils.video_utils.VideoWriter` with `save_video=True`, append observations or frames, and save the video. Camera observations are available without this extra, so you may use your own encoder instead.

## Attribution

This independent package preserves the upstream license statements: original code is MIT-licensed, the upstream dataset is labeled CC BY 4.0, and upstream asset ownership and licensing continue to apply. Cite the [LIBERO paper](https://arxiv.org/abs/2306.03310) and [LIBERO-PRO paper](https://arxiv.org/abs/2510.03827) when using the benchmark in research.
