Metadata-Version: 2.4
Name: mm-evalkit
Version: 0.1.0
Summary: Benchmark MuscleMimic policy kinematics, kinetics, and neuromuscular behavior against human data
Author: MuscleMimic Team
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/amathislab/mm-evalkit
Project-URL: Repository, https://github.com/amathislab/mm-evalkit
Project-URL: Issues, https://github.com/amathislab/mm-evalkit/issues
Project-URL: Preprint, https://arxiv.org/abs/2603.25544
Keywords: biomechanics,benchmarking,electromyography,kinematics,kinetics,musculoskeletal simulation,neuromuscular behavior
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: matplotlib>=3.8
Requires-Dist: mujoco>=3.3
Requires-Dist: musclemimic-models>=1.0.2
Requires-Dist: numpy>=2.0
Requires-Dist: pandas>=2.2
Requires-Dist: pyyaml>=6.0
Requires-Dist: scipy>=1.14
Provides-Extra: render
Requires-Dist: imageio-ffmpeg>=0.6; extra == "render"
Requires-Dist: opencv-python>=4.10; extra == "render"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: imageio-ffmpeg>=0.6; extra == "dev"
Requires-Dist: opencv-python>=4.10; extra == "dev"
Requires-Dist: pre-commit<4,>=3.8; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.9; extra == "dev"
Dynamic: license-file

# MM-EvalKit (MuscleMimic Evaluation Kit)

<p align="center">
  <a href="https://pypi.org/project/mm-evalkit/"><img src="https://img.shields.io/pypi/v/mm-evalkit.svg" alt="PyPI version"></a>
  <a href="https://github.com/amathislab/mm-evalkit/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-76B900.svg" alt="License"></a>
  <a href="https://arxiv.org/abs/2603.25544"><img src="https://img.shields.io/badge/Preprint-arXiv-b31b1b" alt="Preprint"></a>
</p>

`mm-evalkit` is the evaluation benchmark for MuscleMimic policies. It compares
rollout NPZ files with recorded human data at three physiological levels:

- kinematics: joint-angle RMSE and waveform correlation;
- kinetics: joint-moment and vertical-GRF RMSE and waveform correlation; and
- neuromuscular behavior: muscle-activation correlation and muscle RSA against
  human EMG.

The benchmark produces population and subject-matched metrics, human reference
baselines, statistical reports, figures, and exclusion audits.

## Supported datasets

| Dataset | Evaluation | Analysis unit |
|---|---|---|
| Gait120 | Joint angles, joint moments, vertical GRF, muscle activation, and EMG | Subject and gait cycle |
| Wang | Joint angles, muscle activation, and EMG | Population gait profile |
| ULTra-MoCap | Arm kinematics, muscle activation, and EMG | Subject, movement, and trial |

Gait120 provides all three evaluation levels over one retained subject cohort.

## Install

The PyPI distribution is named `mm-evalkit`; it provides the `mm_evalkit` Python
package and the `mm-evalkit` command-line interface. Install it with Python 3.11 or
newer:

```bash
python -m pip install mm-evalkit
```

Synchronized visualization requires the `render` extra:

```bash
python -m pip install 'mm-evalkit[render]'
```

## Configure data paths

```bash
mm-evalkit paths set gait120 /absolute/path/to/Gait120
mm-evalkit paths set wang /absolute/path/to/Wang
mm-evalkit paths set ultra-mocap /absolute/path/to/ULTra-MoCap
mm-evalkit paths show
```

An explicit `--dataset_dir` on an evaluation command overrides the saved path.

## Define policies

Evaluation commands accept a YAML file containing one or more rollout exports:

```yaml
policies:
  - id: policy-a
    label: "Policy A"
    trajectory_data_path: trajectory_data/policy-a/episodes.npz
    color: "#2a78d6"
    linestyle: "-"
```

The rollout `motion_group` and `traj_id` values determine subject and trial
identity. Compared policies must use the same ordered references.

## Evaluate

### Gait120

```bash
mm-evalkit evaluate gait120 \
  --motion_group GAIT120_TESTING \
  --policy_config_file policies.yaml \
  --plots_dir results/gait120 \
  --kinetics_reference_dir /path/to/gait120-kinetics
```

`--kinetics_reference_dir` enables the kinetics layer. The rollout NPZ provides
policy joint-force and foot-contact-force signals. Human joint moments come
from inverse dynamics; human GRF comes from corrected Gait120 force-plate
measurements.

### Wang

```bash
mm-evalkit evaluate wang \
  --policy_config_file policies.yaml \
  --plots_dir results/wang
```

### ULTra-MoCap

```bash
mm-evalkit evaluate ultra-mocap \
  --motion_group ULTRA_MOCAP_BIMANUAL_TESTING_SUBJECTS \
  --policy_config_file policies.yaml \
  --plots_dir results/ultramocap
```

Each run requires an empty output directory. Dataset-specific options are
available through:

```bash
mm-evalkit evaluate --list-datasets
mm-evalkit evaluate gait120 --help
mm-evalkit evaluate wang --help
mm-evalkit evaluate ultra-mocap --help
```

## Outputs

Evaluation outputs include:

- population and subject-matched metric tables;
- waveform and summary figures;
- paired Wilcoxon reports with Holm correction;
- retained-subject and exclusion audits; and
- run manifests where the workflow provides them.

Gait120 kinetics writes `kinetics_profiles.svg`, `kinetics_metrics.svg`,
`kinetics_metrics.csv`, and `kinetics_input_omissions.csv`.

Each dataset command documents its complete configuration through `--help`.
Input archives are validated before analysis starts.

## Synchronized visualization

Renderers synchronize policy motion with human and artificial muscle signals:

```bash
mm-evalkit render gait120 --help
mm-evalkit render ultra-mocap --help
```

## Demos

### Overhead reach (`P11_OverheadReach_180`)

![ULTra-MoCap overhead-reach demo](https://raw.githubusercontent.com/amathislab/mm-evalkit/main/assets/demos/P11_OverheadReach_180_demo.gif)

### Shoulder rotation (`P12_ShoulderRotation_Normal`)

![ULTra-MoCap shoulder-rotation demo](https://raw.githubusercontent.com/amathislab/mm-evalkit/main/assets/demos/P12_ShoulderRotation_Normal_demo.gif)

### Gait step (`S052_step01`)

![Gait120 step demo](https://raw.githubusercontent.com/amathislab/mm-evalkit/main/assets/demos/S052_step01_demo.gif)

## Add a dataset

Dataset adapters declare their source contract, workflow, CLI entrypoint, and
tests. Keep dataset files outside the repository and document their license and
access requirements.

External packages register adapters in `pyproject.toml`:

```toml
[project.entry-points."mm_evalkit.datasets"]
example = "example_package.cli:main"

[project.entry-points."mm_evalkit.renderers"]
example = "example_package.render:prepare_clip"
```

Validation should cover a minimal valid dataset, malformed inputs, alignment,
normalization, metrics, registry discovery, and an end-to-end run. The
[architecture guide](https://github.com/amathislab/mm-evalkit/blob/main/ARCHITECTURE.md)
defines the extension boundaries.

## Development

```bash
git clone https://github.com/amathislab/mm-evalkit.git
cd mm-evalkit
make install-dev
make format
make lint
make test
make precommit
```

Ruff and pre-commit cover every Python file in the repository.

See [Architecture](https://github.com/amathislab/mm-evalkit/blob/main/ARCHITECTURE.md)
for data flow, module ownership, input contracts, and public extension points.

## Citation

```bibtex
@article{Li2026MuscleMimic,
  title={Towards Embodied AI with MuscleMimic:
    Unlocking full-body musculoskeletal motor learning at scale},
  author={Li, Chengkun and Wang, Cheryl and Ziliotto, Bianca and
    Simos, Merkourios and Kovecses, Jozsef and Durandau, Guillaume and
    Mathis, Alexander},
  journal={arXiv preprint arXiv:2603.25544},
  year={2026}
}
```
