Metadata-Version: 2.5
Name: libero-pro-benchmark
Version: 0.1.0
Import-Name: libero
Import-Name: libero_pro_benchmark
Import-Name: perturbation
Summary: Standalone NumPy-based LIBERO and LIBERO-PRO simulation benchmark
License-Expression: MIT
License-File: LICENSE
Requires-Python: <3.15,>=3.10
Requires-Dist: bddl==1.0.1
Requires-Dist: cloudpickle<4,>=3
Requires-Dist: easydict<2,>=1.13
Requires-Dist: future<2,>=1.0
Requires-Dist: gymnasium<2,>=1.0
Requires-Dist: libero-pro-benchmark-assets-1==0.1.0
Requires-Dist: libero-pro-benchmark-assets-2==0.1.0
Requires-Dist: libero-pro-benchmark-assets-3==0.1.0
Requires-Dist: libero-pro-benchmark-assets-4==0.1.0
Requires-Dist: libero-pro-benchmark-assets-5==0.1.0
Requires-Dist: matplotlib<4,>=3.8
Requires-Dist: mujoco<4,>=2.3.7
Requires-Dist: numpy<3,>=1.26
Requires-Dist: opencv-python<5,>=4.8
Requires-Dist: pyyaml<7,>=6
Requires-Dist: robosuite==1.4.0
Requires-Dist: termcolor<4,>=2.4
Requires-Dist: tqdm<5,>=4.66
Provides-Extra: compatible
Requires-Dist: mujoco==2.3.7; extra == 'compatible'
Requires-Dist: numpy==1.26.4; extra == 'compatible'
Requires-Dist: opencv-python<4.12; extra == 'compatible'
Provides-Extra: datasets
Requires-Dist: h5py<4,>=3.11; extra == 'datasets'
Provides-Extra: video
Requires-Dist: imageio-ffmpeg<1,>=0.5; extra == 'video'
Requires-Dist: imageio<3,>=2.34; extra == 'video'
Description-Content-Type: text/markdown

# LIBERO-PRO-Benchmark

A modern, PyTorch-free benchmark for LIBERO and LIBERO-PRO VLA evaluation, with uv setup, original benchmark APIs, and lossless initial-state conversion.

A standalone LIBERO-PRO simulation and task package managed with uv. It provides environments, tasks, success predicates, initial states, and perturbation tools for external policy evaluation. Policy inference and training belong in the caller's project. Benchmark execution does not require PyTorch or CPU/CUDA extras.

**One installation supports both LIBERO and LIBERO-PRO evaluation.** This package includes all tasks in the five standard LIBERO suites (`libero_spatial`, `libero_object`, `libero_goal`, `libero_90`, and `libero_10`), with their task names and base ordering preserved, alongside the LIBERO-PRO benchmark extensions. You do not need to install the original LIBERO and LIBERO-PRO projects separately.

The package is based on upstream commit `eafdb809426b13153aa1e4c42d6601844217dfec`. It preserves all 83 registered suites, 960 task entries, and their original ordering.

This package bundles the official evaluation resources that LIBERO-PRO publishes separately, so users do not need to download and copy them manually. The bundled additions comprise 160 BDDL task files and 160 initialization files for tasks already registered upstream; **no new tasks were added**, and existing files were not overwritten. Initialization arrays are packaged as losslessly converted NPZ files.

The five standard LIBERO suites and all 16 PRO suites with `_lan`, `_object`, `_swap`, or `_task` suffixes have complete resources in the audited inventory. **Remaining gaps concern other registered upstream extensions whose required resources were not supplied by either the source checkout or the official dataset at the recorded revisions.** These extensions cannot run as supplied; this package does not invent replacement resources. See the [resource inventory](docs/provenance/resource-availability.json) and [validation scope](docs/reports/README.md).

## Installation

```bash
cd LIBERO-PRO-Benchmark
uv sync
uv run libero-pro list
uv run libero-pro smoke --no-render
uv run libero-pro smoke
```

Dependency ranges are defined in `pyproject.toml`; exact resolved versions are recorded in `uv.lock`. `uv sync` installs the committed resolution and does not automatically adopt newly released dependencies. To update to newer versions within the declared constraints:

```bash
uv lock --upgrade
uv sync
```

Run the [manual validation checks](docs/validation.md) after an upgrade. The version numbers in [validation reports](docs/reports/README.md) describe measured dependency snapshots, not a promise about future releases. `.python-version` sets the default interpreter; `requires-python` declares the supported range. Use a recent uv release that recognizes the interpreter specified by this checkout.

No `--extra mac`, `--extra linux`, or `--extra cuda` is needed.

### Compatibility baseline

```bash
uv sync --python 3.11 --no-group current --extra compatible
uv run --python 3.11 --no-group current --extra compatible libero-pro smoke
```

`compatible` deliberately pins MuJoCo 2.3.7, NumPy 1.26.4, and OpenCV <4.12. Use Python 3.10/3.11: the old MuJoCo release has no suitable prebuilt wheels for newer Python versions. These are a measured reference baseline, not a claim that arbitrary historical environments or different machines produce identical trajectories.

Keep `--no-group current --extra compatible` on subsequent `uv run` commands; otherwise uv restores the default dependency selection. To retain a separate baseline environment:

```bash
UV_PROJECT_ENVIRONMENT=.venv-compatible uv sync --python 3.11 --no-group current --extra compatible
UV_PROJECT_ENVIRONMENT=.venv-compatible uv run --python 3.11 --no-group current --extra compatible libero-pro smoke
```

To switch back to the default interpreter and dependencies, use `uv sync --python "$(cat .python-version)"`.

### Optional tools

| Extra | Dependencies | API and purpose |
|---|---|---|
| `compatible` | Fixed reference simulation dependencies | Compare against the measured upstream baseline |
| `datasets` | h5py | `libero.libero.utils.dataset_utils.get_dataset_info`: inspect downloaded HDF5 demonstration files |
| `video` | imageio, imageio-ffmpeg | `libero.libero.utils.video_utils.VideoWriter`: explicitly collect frames and encode MP4 videos |

```bash
uv sync --extra datasets --extra video
```

Installing `datasets` does not download demonstrations or add a policy training framework. It is unnecessary for task BDDL files and NPZ initial states. Installing `video` does not enable automatic recording: the caller must create a writer with `save_video=True`, append frames, and call `save()` or exit its context manager. The caller may also save observation images with their own video library. See [complete API examples](docs/optional-tools.md).

The default local dependency group `current` supplies newer simulation constraints to uv's universal lockfile. It adds no new package names and is mutually exclusive with `compatible`. Published library requirements allow both dependency stacks; the caller's project controls its own resolution. robosuite and BDDL remain pinned to preserve their original APIs. The package has no notebooks; install Jupyter in your own project when needed.

## Python API

Existing VLA evaluation code can retain the original benchmark imports:

```python
from pathlib import Path
import numpy as np
from libero.libero import benchmark, get_libero_path
from libero.libero.envs import OffScreenRenderEnv

suite = benchmark.get_benchmark("libero_spatial")()
task = suite.get_task(0)
states = suite.get_task_init_states(0)  # ndarray with the original dtype and shape
bddl = Path(get_libero_path("bddl_files")) / task.problem_folder / task.bddl_file

env = OffScreenRenderEnv(bddl_file_name=str(bddl), camera_heights=128, camera_widths=128)
try:
    env.seed(123)
    env.reset()
    obs = env.set_init_state(states[0])
    obs, reward, done, info = env.step(np.zeros(env.env.action_dim))
    success = env.check_success()
finally:
    env.close()
```

The alternate entry points `from libero_pro_benchmark import benchmark, get_libero_path` and `from libero_pro_benchmark.envs import OffScreenRenderEnv` refer to the same benchmark module and environment class. Original observation keys, 7-dimensional Panda actions, and the four-item `step()` result are preserved. `SegmentationRenderEnv`, `DummyVectorEnv`, and `SubprocVectorEnv` also retain their original import paths.

### Distribution and import names

The installable distribution is named `libero-pro-benchmark`. It provides three top-level Python import names:

- `libero`: the original benchmark namespace, including `libero.libero`.
- `libero_pro_benchmark`: an additional entry point and package-specific tools.
- `perturbation`: the original top-level perturbation compatibility module.

`project.import-names` declares these names to packaging tools; actual modules implement the imports. It does not create aliases by itself, choose a device, or install multiple distributions.

Remove the upstream distribution from the caller's dependencies before adding this package. Install the main package from PyPI; all five required resource packages are installed automatically:

```bash
pip install libero-pro-benchmark
# Or, in a uv project:
uv add libero-pro-benchmark
```

For checkout development, run `uv sync` in this repository. To test built distributions in another project before uploading them, build all six distributions and supply the local resource wheels:

```bash
uv remove libero  # Only if the caller's project declares the upstream dependency.
uv add --find-links /absolute/path/to/LIBERO-PRO-Benchmark/dist/release \
  /absolute/path/to/LIBERO-PRO-Benchmark/dist/release/libero_pro_benchmark-0.1.0-py3-none-any.whl
```

On import, this package rejects installed `libero`, `libero-pro`, or `libero-ood` distributions, and other distributions declaring an overlapping `Import-Name`. Its own import metadata helps supporting installers detect overlaps. The upstream package lacks that metadata and installer behavior varies, so installation-time mutual exclusion cannot be guaranteed. PyTorch used by an external policy is allowed.

The optional `libero-pro` console script lists tasks, audits resources, and runs simulation smoke checks. VLA evaluation can use the Python API directly.

### Initial states and configuration

Task metadata retains the original `.pruned_init` filename; packaged data is stored as `.pruned_init.npz`. Read it through `suite.get_task_init_states()` or `libero_pro_benchmark.states.load_initial_states()`. External code calling `torch.load(path)` directly must change its loading call.

Default task and initial-state paths are resolved from the main package; `get_libero_path("assets")` points to the complete installed resource tree. There are no import-time prompts, configuration writes, resource copies, or runtime downloads. Explicit setters use `~/.config/libero-pro-benchmark`; the old implicit `~/.libero` configuration is not read. Set `LIBERO_CONFIG_PATH=/your/config-dir` to use a `config.yaml` containing resource path overrides. Initial-state overrides must point to NumPy-format data. `LIBERO_DATASETS_PATH` separately overrides the demonstration directory.

### Perturbation and state generation

The five upstream perturbation types are preserved in `libero_pro_benchmark.perturbation`, with the original `import perturbation` entry point. Both expose the same class and function objects. Bundled YAML and templates are available through `importlib.resources.files("libero_pro_benchmark")`. See [perturbation examples and equivalence limits](docs/optional-tools.md#perturbation).

```bash
uv run python -m libero_pro_benchmark.generate_init_states \
  --bddl_base_dir /path/to/bddl \
  --output_dir /path/to/init_files --num_inits 50
```

The generator retains the upstream sampling process and writes NPZ arrays. Resampled states are not substitutes for official evaluation states. The upstream `ood_object.yaml` has an invalid empty key; this package repairs it to `wooden_cabinet` and retains the original configuration as a test fixture. Upstream comments, docstrings, and diagnostics are retained as supplied.

### Compatibility scope

The preserved benchmark definitions and signatures cover 58 modules, 85 module-level functions, and 271 classes, including their methods. Tests also compare task metadata/order and exercise original path-helper keywords such as `get_path_dict(root_location=...)` and `utils.get_libero_path(key=...)`.

Compatibility covers the benchmark Python API. NPZ storage, configuration location, resource package location, noninteractive imports, and removal of the training subsystem are explicit differences. Use `get_libero_path("assets")` instead of constructing paths from module `__file__` values. The original repository's training code and maintenance commands are outside this package. Matching APIs do not imply identical physics across dependency versions.

## Data integrity and resource coverage

- All 848 initial-state arrays (688 upstream files and 160 official additions) were converted losslessly. Their dtype, shape, count, ordering, and C-order byte SHA256 were checked. No float32 conversion is performed. The original `torch.save()` files contained NumPy arrays, not Tensor storage.
- Runtime loading uses `np.load(..., allow_pickle=False)`. The offline converter supports the original NumPy archives without PyTorch. Its explicit `--allow-torch` option is for a separate migration environment containing real dense tensors; unsupported dtypes fail rather than being cast.
- Original assets and BDDL retain their source-content fingerprints. Upstream comments and diagnostics are retained; benchmark code includes the documented storage, configuration, optional-import, and simulator API adaptations. Success predicates and controller formulas are preserved.
- **Default dependencies allow physics and rendering differences from the old baseline.** Compatibility-mode bitwise equality has been measured for specific samples; it is not guaranteed for all trajectories, machines, or historical environments.

Of 960 registered task entries, 612 have BDDL and nonempty initial states; 409 also have the known required model assets. All entries in 33 suites have resources, and the first task of each suite passed reset, state restoration, and stepping. Remaining gaps include missing BDDL/states, empty state files, and upstream models `red_sticker`, `blue_red_sticker`, `red_box`, and `libero_mug_green`. Models available in the upstream notebooks, including `libero_mug_yellow`, are packaged. Resource availability is not a policy success-rate certification.

```bash
uv run libero-pro audit --output docs/reports/runs/resource-audit.json
uv run libero-pro audit --strict  # Returns nonzero while upstream resource gaps remain.
```

Official additions come from [LIBERO-Pro on Hugging Face](https://huggingface.co/datasets/zhouxueyang/LIBERO-Pro), pinned to revision `c86fc3b8293185a6f373677018ff3e37f8391602`. Their paths and hashes are recorded in [official resource provenance](docs/provenance/official-resources.json).

## macOS and Linux

Native macOS OpenGL offscreen rendering has been tested on Apple Silicon in both dependency modes. Do not set `MUJOCO_GL=egl` on macOS. Interactive MuJoCo viewers have separate `mjpython` requirements.

On Linux with EGL and a suitable GPU driver:

```bash
uv sync
MUJOCO_GL=egl uv run libero-pro smoke
```

For CPU software rendering with system OSMesa libraries:

```bash
MUJOCO_GL=osmesa uv run libero-pro smoke
```

The system must supply the selected OpenGL backend and drivers. Simulation rendering and policy inference devices are independent settings. Linux awaits testing on a real machine; no CI or scheduled automation has been added. See [manual validation commands](docs/validation.md).

One upstream scene XML filename contained a DEL control character rejected by current archive tools. The packaged filename is normalized to `yellow_book.xml`, with unchanged contents and an explicit [path mapping](docs/provenance/path-migrations.json). The original object class uses a different model directory.

## Validation and publication

[docs/reports/README.md](docs/reports/README.md) summarizes the principal findings and limitations. Git tracks compact reports, provenance, original API/task fixtures, tests, reproduction tools, and `uv.lock`. Detailed local outputs belong in ignored `docs/reports/runs/`; regression tests do not depend on them. Exact versions in the reports are historical measurements and must remain attached to their results after dependency upgrades.

```bash
uv sync --group dev
uv run --group dev pytest -q
uv run python tools/build_packages.py
```

The main distribution contains benchmark code, task BDDL files, initial-state arrays, configurations, and templates. Five mandatory `libero-pro-benchmark-assets-1` through `-5` distributions contain the original XML, meshes, and textures. A single `uv sync` installs all of them; no resource extra or first-use download is needed. The resource packages use non-editable local path dependencies, while the main package remains editable.

The former single wheel was about 345 MiB. The split distributes the same resource bytes across five smaller wheels and leaves the main wheel about 5 MiB. Every wheel and source archive must pass the 100,000,000-byte release check before publication. Run `uv build` to build only the main distribution; use `tools/build_packages.py` to build all six. Publish the resource distributions before the main distribution. Local reports are excluded from source archives and Git. See [package layout, installation, and manual release checks](docs/packaging.md) and the [validated split results](docs/reports/package-split.md).

## Attribution and license

This is an independent extraction, not an official release. Original code retains the [MIT license](LICENSE); the upstream dataset is labeled [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Upstream asset ownership and licensing continue to apply. Cite [LIBERO](https://arxiv.org/abs/2306.03310) and [LIBERO-PRO](https://arxiv.org/abs/2510.03827) when using the benchmark in research.
