Metadata-Version: 2.4
Name: praxis-eval
Version: 0.1.2
Summary: Standalone robot-policy evaluation harness with local and remote policy adapters.
Home-page: https://chaoqi-liu.com
Author: Chaoqi Liu
Author-email: liuchaoqi730@gmail.com
Maintainer: Chaoqi Liu
Maintainer-email: liuchaoqi730@gmail.com
License-Expression: Apache-2.0
Project-URL: Homepage, https://chaoqi-liu.com
Project-URL: Source, https://github.com/Chaoqi-LIU/praxis-eval
Project-URL: Issues, https://github.com/Chaoqi-LIU/praxis-eval/issues
Project-URL: Documentation, https://chaoqi-liu.com/praxis-eval/
Keywords: robotics,evaluation,simulation,policy,benchmark
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: numpy>=1.26.0
Requires-Dist: omegaconf>=2.3.0
Provides-Extra: dev
Requires-Dist: gymnasium>=1.0.0; extra == "dev"
Requires-Dist: lerobot==0.4.4; extra == "dev"
Requires-Dist: omegaconf>=2.3.0; extra == "dev"
Requires-Dist: pre-commit>=4.0.0; extra == "dev"
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: praxis-remote<0.2.0,>=0.1.0; extra == "dev"
Requires-Dist: praxis-simpler==0.0.1.post2; extra == "dev"
Requires-Dist: praxis-mshab==0.0.1.post2; extra == "dev"
Requires-Dist: pyyaml>=6.0.0; extra == "dev"
Requires-Dist: ruff>=0.4.0; extra == "dev"
Requires-Dist: torch>=2.7.0; extra == "dev"
Requires-Dist: tqdm>=4.66.0; extra == "dev"
Provides-Extra: remote
Requires-Dist: praxis-remote<0.2.0,>=0.1.0; extra == "remote"
Provides-Extra: libero
Requires-Dist: gymnasium>=1.0.0; extra == "libero"
Requires-Dist: lerobot==0.4.4; extra == "libero"
Requires-Dist: omegaconf>=2.3.0; extra == "libero"
Requires-Dist: pyyaml>=6.0.0; extra == "libero"
Requires-Dist: praxis-robosuite==1.5.2; extra == "libero"
Requires-Dist: praxis-libero==0.1.1; extra == "libero"
Requires-Dist: torch>=2.7.0; extra == "libero"
Requires-Dist: tqdm>=4.66.0; extra == "libero"
Provides-Extra: metaworld
Requires-Dist: gymnasium>=1.0.0; extra == "metaworld"
Requires-Dist: lerobot[metaworld]==0.4.4; extra == "metaworld"
Requires-Dist: omegaconf>=2.3.0; extra == "metaworld"
Requires-Dist: torch>=2.7.0; extra == "metaworld"
Requires-Dist: tqdm>=4.66.0; extra == "metaworld"
Provides-Extra: robocasa
Requires-Dist: gymnasium>=1.0.0; extra == "robocasa"
Requires-Dist: lerobot==0.4.4; extra == "robocasa"
Requires-Dist: omegaconf>=2.3.0; extra == "robocasa"
Requires-Dist: numpy==2.2.5; extra == "robocasa"
Requires-Dist: mujoco==3.3.1; extra == "robocasa"
Requires-Dist: praxis-robosuite==1.5.2; extra == "robocasa"
Requires-Dist: praxis-robocasa==1.0.0; extra == "robocasa"
Requires-Dist: lxml>=5.0.0; extra == "robocasa"
Requires-Dist: torch>=2.7.0; extra == "robocasa"
Requires-Dist: tqdm>=4.66.0; extra == "robocasa"
Provides-Extra: robomimic
Requires-Dist: gymnasium>=1.0.0; extra == "robomimic"
Requires-Dist: lerobot==0.4.4; extra == "robomimic"
Requires-Dist: omegaconf>=2.3.0; extra == "robomimic"
Requires-Dist: robomimic==0.2.0; extra == "robomimic"
Requires-Dist: praxis-robosuite==1.5.2; extra == "robomimic"
Requires-Dist: torch>=2.7.0; extra == "robomimic"
Requires-Dist: tqdm>=4.66.0; extra == "robomimic"
Provides-Extra: simpler
Requires-Dist: lerobot==0.4.4; extra == "simpler"
Requires-Dist: omegaconf>=2.3.0; extra == "simpler"
Requires-Dist: praxis-remote<0.2.0,>=0.1.0; extra == "simpler"
Requires-Dist: praxis-simpler==0.0.1.post2; extra == "simpler"
Requires-Dist: tqdm>=4.66.0; extra == "simpler"
Provides-Extra: mshab
Requires-Dist: lerobot==0.4.4; extra == "mshab"
Requires-Dist: omegaconf>=2.3.0; extra == "mshab"
Requires-Dist: praxis-remote<0.2.0,>=0.1.0; extra == "mshab"
Requires-Dist: praxis-mshab==0.0.1.post2; extra == "mshab"
Requires-Dist: tqdm>=4.66.0; extra == "mshab"
Provides-Extra: all
Requires-Dist: gymnasium>=1.0.0; extra == "all"
Requires-Dist: lerobot[metaworld]==0.4.4; extra == "all"
Requires-Dist: omegaconf>=2.3.0; extra == "all"
Requires-Dist: praxis-remote<0.2.0,>=0.1.0; extra == "all"
Requires-Dist: pyyaml>=6.0.0; extra == "all"
Requires-Dist: praxis-robosuite==1.5.2; extra == "all"
Requires-Dist: praxis-libero==0.1.1; extra == "all"
Requires-Dist: numpy==2.2.5; extra == "all"
Requires-Dist: mujoco==3.3.1; extra == "all"
Requires-Dist: praxis-robocasa==1.0.0; extra == "all"
Requires-Dist: lxml>=5.0.0; extra == "all"
Requires-Dist: robomimic==0.2.0; extra == "all"
Requires-Dist: praxis-simpler==0.0.1.post2; extra == "all"
Requires-Dist: praxis-mshab==0.0.1.post2; extra == "all"
Requires-Dist: torch>=2.7.0; extra == "all"
Requires-Dist: tqdm>=4.66.0; extra == "all"
Dynamic: license-file

# praxis-eval

`praxis-eval` is a standalone robot-policy evaluation package for simulation benchmarks. It owns benchmark setup, rollout execution, metrics, artifacts, and observation/action contracts; users provide a policy adapter that consumes documented observations and returns documented actions.

## Installation

`praxis-eval` supports Python 3.10 and newer.

```bash
pip install praxis-eval
```

Install the benchmark extras you plan to run:

```bash
pip install "praxis-eval[libero]"
pip install "praxis-eval[robocasa]"
pip install "praxis-eval[robomimic]"
pip install "praxis-eval[metaworld]"
pip install "praxis-eval[simpler]"
pip install "praxis-eval[mshab]"
pip install "praxis-eval[remote]"
```

## Minimal Local Evaluation

This example assumes the LIBERO extra and simulator runtime are installed.

```python
import numpy as np

from praxis_eval import EvalConfig, LocalPolicy, get_driver, evaluate


class RandomPolicy:
    def __init__(self, seed: int = 0) -> None:
        self.rng = np.random.default_rng(seed)

    def reset(self, episode_ids=None) -> None:
        pass

    def act(self, observations, *, action_spec=None, policy_kwargs=None, episode_ids=None):
        del policy_kwargs, episode_ids
        if action_spec is None or action_spec.shape is None:
            raise ValueError("This example needs a benchmark ActionSpec.")
        low = -1.0 if action_spec.minimum is None else action_spec.minimum
        high = 1.0 if action_spec.maximum is None else action_spec.maximum
        return self.rng.uniform(
            low=low,
            high=high,
            size=(len(observations), *action_spec.shape),
        ).astype(action_spec.dtype)


driver = get_driver("libero")
print(driver.contract)

result = evaluate(
    "libero",
    policy=LocalPolicy(RandomPolicy(seed=42)),
    config=EvalConfig(
        task="libero_10",
        task_ids=(0,),
        num_eval_per_task=1,
        num_parallel_env=1,
        output_dir="eval/libero_smoke",
    ),
)

print(result.overall)
print(result.artifacts)
```

## Supported Benchmarks

| Driver | Install extra | Runtime model |
| --- | --- | --- |
| `libero` | `praxis-eval[libero]` | Runs in the current Python environment. |
| `robocasa` | `praxis-eval[robocasa]` | Runs in the current Python environment after asset setup. |
| `robomimic` | `praxis-eval[robomimic]` | Runs in the current Python environment. |
| `metaworld` | `praxis-eval[metaworld]` | Runs in the current Python environment. |
| `simpler` | `praxis-eval[simpler]` | Uses a dedicated SimplerEnv runtime for simulator execution. |
| `mshab` | `praxis-eval[mshab]` | Uses a dedicated MS-HAB runtime for simulator execution. |

Use the setup and verification CLIs to inspect available benchmark-specific commands:

```bash
praxis-eval-setup --help
praxis-eval-verify --help
```

## Documentation

Full documentation is available at <https://chaoqi-liu.com/praxis-eval/>, including installation details, benchmark contracts, remote policy evaluation, runtime setup, examples, and the developer guide for adding new benchmarks.

## License And Citation

`praxis-eval` is licensed under Apache-2.0. If this package supports your research or product work, cite the repository using [CITATION.cff](CITATION.cff).
