Metadata-Version: 2.4
Name: magma_bench
Version: 2.0.1
Summary: Evaluate agents on interactive robotic tasks with MAGMA
Project-URL: Documentation, https://magma-rob.github.io/docs/intro
Project-URL: Repository, https://github.com/MAGMA-rob/magma-bench
Project-URL: Issues, https://github.com/MAGMA-rob/magma-bench/issues
Project-URL: Changelog, https://github.com/MAGMA-rob/magma-bench/blob/main/CHANGELOG.md
Classifier: Development Status :: 5 - Production/Stable
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: huggingface_hub<2,>=1.4
Requires-Dist: magma_core[simulation]<3.0.0,>=2.0.0
Requires-Dist: magma_scenarios<3.0.0,>=2.1.0
Requires-Dist: packaging<27,>=24
Requires-Dist: pydantic<3,>=2.4
Requires-Dist: PyYAML<7,>=6.0
Requires-Dist: requests<3,>=2.34.2
Requires-Dist: torch<3,>=2.8
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Provides-Extra: video
Requires-Dist: imageio-ffmpeg>=0.5; extra == "video"
Requires-Dist: numpy>=1.23; extra == "video"
Requires-Dist: Pillow>=10; extra == "video"
Dynamic: license-file

# MAGMA-BENCH

Evaluate agents on interactive robotic tasks with MAGMA. MAGMA-BENCH loads
compiled benchmark episodes, runs them in simulation, communicates with an agent
through the MAGMA HTTP protocol, and saves detailed results and metrics.

**Version 2.0.1 adds versioned benchmark downloads.** Dataset releases have
independent versions; MAGMA-BENCH dataset 1.0.0 requires MAGMA Bench 2.0.1,
Core 2.0.1, and Scenarios 2.1.1 exactly.

[Official documentation](https://magma-rob.github.io/docs/intro) ·
[Report an issue](https://github.com/MAGMA-rob/magma-bench/issues)

## Installation

Python 3.12 is required. Simulation installation has been checked on Linux
x86_64. GPU simulation and rendering require compatible system drivers.

```bash
python -m pip install "magma_bench==2.0.1"
```

This installs `magma_core[simulation]>=2.0.0,<3.0.0` and
`magma_scenarios>=2.1.0,<3.0.0` with their Python dependencies. The benchmark
files and the agent server are separate inputs and are not bundled with this
package.

To upgrade to the latest stable release, use:

```bash
python -m pip install --upgrade magma_bench
```

## Usage

Download the latest stable official dataset, or request an exact release:

```bash
magma-bench fetch
magma-bench fetch --version 1.0.0
```

Releases come from [lbernat/MAGMA-BENCH](https://huggingface.co/datasets/lbernat/MAGMA-BENCH).
The default destination is `$XDG_CACHE_HOME/magma-bench/datasets/v1.0.0/`,
or `~/.cache/magma-bench/datasets/v1.0.0/` when `XDG_CACHE_HOME` is unset.
Each download records its immutable Hub commit and file checksums. A published
tag must never be moved. Repeating a fetch reuses a complete installation;
a changed tag or damaged installation is rejected. To recover, choose a new
empty `--path` or remove the damaged cache directory before fetching again.

Start a compatible MAGMA agent server, then run the downloaded release offline:

```bash
magma-bench run \
  --dataset 1.0.0 \
  --results-path ./eval/experiment-1 \
  --agent-address http://127.0.0.1:8888 \
  --run-name experiment-1
```

`run` never downloads data. `--dataset` accepts an exact version, with or without
`v`, and looks only in the conventional cache. Package versions are checked
before runtime initialization; a mismatch prints the exact installation command.
Different dataset versions can coexist on disk, but running them may require
separate Python environments for their different package pins.

To download to a custom destination, use:

```bash
magma-bench fetch --version 1.0.0 --path ./compiled-benchmark
```

Run that directory, or your own compiled benchmark, with `--benchmark-root`:

```bash
magma-bench run \
  --benchmark-root /path/to/compiled-benchmark \
  --results-path /path/to/results/experiment-1 \
  --agent-address http://127.0.0.1:8888 \
  --run-name experiment-1
```

`--dataset` and `--benchmark-root` are mutually exclusive; one is required.
Every benchmark directory, including a custom build, must contain both
`benchmark.json` and a release `manifest.json`:

```json
{
  "schema_version": "1.0",
  "benchmark_version": "1.0.0",
  "requirements": {
    "magma_bench": "2.0.1",
    "magma_core": "2.0.1",
    "magma_scenarios": "2.1.1"
  }
}
```

Use the same `benchmark_version` in both files and exact package versions in
`requirements`. The compiled artifact schema remains `1.8`; the distribution
manifest has its own schema `1.0`. Install the official release environment with:

```bash
python -m pip install "magma_bench==2.0.1" "magma_core[simulation]==2.0.1" "magma_scenarios==2.1.1"
```

Add the manifest to older custom builds before using this CLI. Its contents
participate in result fingerprints, so changing pins requires a new result
directory. Old `draft` evaluations are development results; start fresh for the
published dataset 1.0.0. `stats` checks manifest consistency without requiring the
pinned runtime packages.

The agent must expose `/health`, `/v1/info`, and `/v1/responses` using protocol
version 2.0. Configuration can be supplied with `--config-path`; command-line
options override its benchmark settings.

Useful options include:

```bash
magma-bench run --help
magma-bench run --benchmark-root /path/to/benchmark --results-path ./eval/coffee --scenarios coffee_comp
magma-bench run --benchmark-root /path/to/benchmark --results-path ./eval/no-judge --skip-judge
magma-bench run --benchmark-root /path/to/benchmark --results-path ./eval/logged --model-logs --videos all
magma-bench run --benchmark-root /path/to/benchmark --results-path ./eval/failures --videos planner-failure
```

Video output uses the optional dependencies:

```bash
python -m pip install "magma_bench[video]==2.0.1"
```

Merge all completed parts of one evaluation into a YAML analysis:

```bash
magma-bench stats ./stats.yaml ./eval/model-name /path/to/compiled-benchmark
```

The results directory must contain one subdirectory per part, each with `run.json`
and `result.json`. All compiled episodes must be covered exactly once. Partial
runs and infrastructure failures stop the command without writing the YAML.
The `name` field is the results directory name. The file includes success rates,
conditional robustness, failure statuses, and scenario, skeleton, semantic,
length, and track breakdowns. Stage progression is the mean of each episode's
fraction of completed stages, using stage counts from the compiled benchmark.
A benchmark fingerprint mismatch is reported in the YAML and on stderr because
episode IDs alone cannot certify that compiled contents match exactly.

Version 2 uses result schema `3.0`. Terminal statuses are `success`,
`invalid_format`, `textual_hallucination`, `non_requested_action`,
`non_physical_exceeded`, `judge_failure`, `env_failure`, `budget_exceeded`,
and `infrastructure_failure`. An infrastructure failure returns the episode to
the end of its current simulator group, for at most three attempts per launch.
The progress log records every attempt; episode results, model logs, and videos
retain the final attempt. Resume a compatible run with `--results-path`; a
resumed infrastructure failure receives a fresh retry allowance. Result schema
`2.0` and older agent adapters are unsupported.

See the [official documentation](https://magma-rob.github.io/docs/intro) for
benchmark preparation, configuration, agent setup, metrics, and result formats.

## Install from source

Install the tagged release from GitHub:

```bash
python -m pip install "magma_bench @ git+https://github.com/MAGMA-rob/magma-bench.git@v2.0.1"
```

Dependencies are resolved from PyPI. For local development, clone the repository
and run `python -m pip install -e ".[dev,video]"`.

## License

[BSD 2-Clause](https://github.com/MAGMA-rob/magma-bench/blob/main/LICENSE).
