Metadata-Version: 2.4
Name: worldevals
Version: 0.3.2
Summary: A curated catalog of physical-AI benchmarks built on Inspect Robots — the Inspect Evals for robotics.
Project-URL: Homepage, https://github.com/robocurve/worldevals
Project-URL: Documentation, https://worldevals.org/
Project-URL: Framework, https://github.com/robocurve/inspect-robots
Author: RoboCurve
License-Expression: MIT
License-File: LICENSE
Keywords: benchmarks,catalog,evaluation,physical-ai,robotics,vla
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: inspect-robots>=0.3
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pre-commit>=3.5; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-gen-files<0.6.1,>=0.5; extra == 'docs'
Requires-Dist: mkdocs-llmstxt>=0.2; extra == 'docs'
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocs>=1.6; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.26; extra == 'docs'
Description-Content-Type: text/markdown

<div align="center">

# WorldEvals

A curated catalog of physical-AI / VLA benchmarks: each one installable,
versioned, and runnable on real robots or simulators via
[Inspect Robots](https://github.com/robocurve/inspect-robots).

If you know [Inspect Evals](https://inspect.aisi.org.uk/evals/), this is that for robotics.

![Status: alpha](https://img.shields.io/badge/status-alpha-blue)
[![CI](https://github.com/robocurve/worldevals/actions/workflows/ci.yml/badge.svg)](https://github.com/robocurve/worldevals/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
[![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen)](https://github.com/robocurve/worldevals/actions/workflows/ci.yml)
[![Docs coverage](https://img.shields.io/badge/public%20docstrings-100%25-brightgreen)](https://github.com/robocurve/worldevals/actions/workflows/ci.yml)
[![Built on Inspect Robots](https://img.shields.io/badge/built%20on-Inspect%20Robots-indigo)](https://github.com/robocurve/inspect-robots)

**[Browse the catalog → worldevals.org](https://worldevals.org/)**

</div>

> **Note:**
> This project is in early development. The API may change between releases, so pin a version before depending on it.

[Inspect Robots](https://github.com/robocurve/inspect-robots) is the framework;
WorldEvals is the collection. Each benchmark lives in its own repository and
owns its release cadence, dependencies, hardware notes, and leaderboard.
WorldEvals is the lightweight index that ties them together: what benchmarks
exist, what tasks each provides, and how to install them. If you come from
Inspect AI, this is the Inspect Evals of that ecosystem, minus the monorepo.

- `inspect-robots list` tells you what's installed.
- `worldevals list` tells you what exists and how to get it.

## Benchmarks

| Benchmark | Tasks | Tags | Status |
|---|--:|---|---|
| [KitchenBench](https://github.com/robocurve/kitchenbench): 10 bimanual kitchen-manipulation tasks | 10 | kitchen, bimanual, manipulation | alpha |

## Install & use

```bash
pip install worldevals   # pulls in inspect-robots

worldevals list                 # all benchmarks
worldevals list --tag bimanual  # filter by tag
worldevals info kitchenbench    # repo, install command, task keys
worldevals tasks                # Inspect Robots tasks installed locally, by benchmark
```

Then install a benchmark and run it through Inspect Robots:

```bash
pip install kitchenbench
inspect-robots run --task kitchenbench/pour_pasta --policy kitchen_scripted --embodiment kitchen
```

## Backends (run on real robots)

Benchmarks are embodiment-agnostic; backend adapters supply a concrete
`Policy` + `Embodiment` so a benchmark runs on real hardware or a simulator.
These are their own repos too (not catalog entries):

| Adapter | Policy · Embodiment | Stack |
|---|---|---|
| [inspect-robots-yam](https://github.com/robocurve/inspect-robots-yam) | `molmoact2` · `yam_arms` | [MolmoAct2](https://github.com/allenai/molmoact2) on [I2RT YAM](https://i2rt.com/products/yam-6-dof-arm) bimanual arms |
| [inspect-robots-so101](https://github.com/robocurve/inspect-robots-so101) | `lerobot` · `so_arm` | [LeRobot](https://github.com/huggingface/lerobot) policies (ACT, SmolVLA, π0, …) on [SO-ARM](https://github.com/TheRobotStudio/SO-ARM100) (SO-100 / SO-101) follower arms |

```bash
inspect-robots run --task kitchenbench/pour_pasta --policy molmoact2 --embodiment yam_arms
```

## Add your benchmark

A benchmark is any repo that:

1. depends on `inspect-robots`,
2. defines one or more Inspect Robots `Task`s, and
3. registers them via `[project.entry-points."inspect_robots.tasks"]` (and, if it ships
   a sim/embodiment or policy, `inspect_robots.embodiments` / `inspect_robots.policies`).

To list it here, add a `Benchmark(...)` entry to
[`src/worldevals/catalog.py`](src/worldevals/catalog.py) and open a PR. A test
validates every entry (unique name, well-formed repo URL, ≥1 task key). See
[KitchenBench](https://github.com/robocurve/kitchenbench) as the reference
implementation.

## Publishing model checkpoints

Benchmarks compare policies, and a benchmark number is only interpretable when the
checkpoint behind it is documented. Every model checkpoint published to the
[robocurve HuggingFace org](https://huggingface.co/robocurve) must follow the
[model-card standard](docs/model-cards.md) (start from the
[template](docs/model-card-template.md); the standard includes a gating checklist).

Published checkpoints are catalogued in [WorldPolicies](https://github.com/robocurve/worldpolicies)
(CI-validated registry + [🤗 Collection](https://huggingface.co/collections/robocurve/worldpolicies-6a4dc8fd556a82aeea0fca37)).

## Development

> **Dependency changes:** after editing dependencies in `pyproject.toml`, run
> `uv lock` and commit the updated lockfile: CI installs with
> `uv sync --locked` and fails with "the lockfile needs to be updated" if you
> forget. Day-to-day conventions (PR-only `main`, the required `ci-ok` check,
> one-click releases) are documented in [`CLAUDE.md`](CLAUDE.md).

Every public module, class, and function needs a docstring, enforced by `ruff` D1;
state the contract, don't restate the name.

```bash
uv venv && uv pip install -e ".[dev]"     # inspect_robots resolved from the v0.3.0 tag
uv run pre-commit install
uv run pytest --cov                        # 100% coverage required
uv run ruff check . && uv run mypy
```

## Citation

If you use WorldEvals in your research, please cite it:

```bibtex
@software{worldevals,
  author  = {Robocurve},
  title   = {WorldEvals: A curated catalog of physical-AI benchmarks},
  year    = {2026},
  url     = {https://github.com/robocurve/worldevals},
  version = {0.3.0},
  license = {MIT}
}
```

## License

[MIT](LICENSE)
