Metadata-Version: 2.4
Name: simready-benchmark
Version: 2026.6.5
Summary: SimReady Benchmark, the engine-agnostic core for validating SimReady USD assets
Author: NVIDIA Corporation
License-Expression: Apache-2.0
Project-URL: Homepage, https://www.nvidia.com
Keywords: nvidia,simready,test,validation
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: filelock<4.0,>=3.13.0
Requires-Dist: usd-core>=23.5
Provides-Extra: kit
Requires-Dist: simready-benchmark-engine-kit; extra == "kit"
Dynamic: license-file

# simready-benchmark

Engine-agnostic Python framework for benchmarking SimReady USD assets.
Discovers applicable tests, executes them across one or more engine sessions
in parallel, and produces HTML and JSON reports.

Part of the [SimReady Python Library Suite](https://developer.nvidia.com/simready)
alongside `simready-validate` and `simready-search`.

---

## Installation

```bash
pip install simready-benchmark          # framework + CLI only
pip install simready-benchmark[kit]     # + Kit / Isaac Sim engine plugin
```

The bundled FET test suite (`simready-benchmark-kit-suite`) is part of the
`simready_foundations` repository. Refer to the install runbook in the
package documentation for the setup.

Python 3.11 or newer is required.

---

## Quick Start

```bash
# Plan + run + report on every asset in the project, in one pass
simready-benchmark --project-config /path/to/project_config.toml

# Or pass paths ad-hoc, scoped to one feature
simready-benchmark --sr-specs /path/to/sr_specs \
              --assets /path/to/asset.usd \
              --features FET003
```

Outputs land in `_testing/` next to the project config (or the current
directory if no config is set):

- `plan.json`. The resolved plan
- `state/events.jsonl`. Live JSON-Lines event stream
- `results/<asset-dir>/<asset>.usd`. Stamped mirror copy
- `results/<asset-dir>/.simready/validation.json`. `runtime_tests[]` receipt
- `results/<asset-dir>/.simready/runtime/<test>/result.json`. Per-test artifacts
- `logs/<engine>_<n>.log`. Per-engine subprocess logs
- `index.html` (HTML report at the output-dir root) plus `report/test_results_index.json`. Aggregated reports

The original asset directory is never modified.

---

## Key Concepts

### Plan / Run / Report Pipeline

A single `simready-benchmark` invocation runs three phases by default:

1. **Plan.** Scan installed test packs and the active sr_specs; match assets to applicable tests; write `plan.json`.
2. **Run.** Execute the plan across one or more engine sessions in parallel; stream JSON events; write per-test `result.json`.
3. **Report.** Aggregate per-test results into HTML and JSON reports.

A final **stamp** phase (default-on, suppress with `--no-stamp`) writes
benchmark metadata into the mirror-copy USDs and merges a `runtime_tests[]`
array into `.simready/validation.json`. Refer to the
[stamp-results skill](src/simready_benchmark/skills/stamp-results/SKILL.md).

### Engines and Test Packs

Engines are registered through the `simready_benchmark.engines` entry-point group;
test packs through `simready_benchmark.tests`. The framework discovers both at
import time. No per-install configuration is needed for installed wheels.

Out of the box, `simready-benchmark` registers an `nvcf` engine plugin for
NVIDIA Cloud Functions execution. `simready-benchmark-engine-kit` adds the `kit` engine
plugin for Kit / Isaac Sim. Third-party engines and test packs slot in cleanly
through the same mechanism. Refer to the
[write-test-pack skill](src/simready_benchmark/skills/write-test-pack/SKILL.md).

### Engine Configuration

Scaffold a starter `engines.toml` and edit the placeholders:

```bash
simready-benchmark --init-engines-toml             # writes ~/.simready-benchmark/engines.toml
simready-benchmark --init-engines-toml ./          # writes ./engines.toml (per-project)
```

The framework reads engine paths from `engines.toml` using a 4-tier lookup
(first match wins):

1. `--engines-toml <FILE>` CLI flag
2. `$SIMREADY_ENGINES_TOML` env var (missing file is a hard error)
3. `engines.toml` in the current working directory
4. `~/.simready-benchmark/engines.toml`

A `[engines.toml] <path>` line goes to stderr one time per process so the user
can identify which file is in effect (suppress with `SIMREADY_QUIET_ENGINES_TOML=1`).

```toml
[kit.isaac_sim]
executable_path = "C:/Program Files/NVIDIA/IsaacSim/2025.1/isaac-sim.bat"
version = "5.1.0"
tags = ["isaac", "kit"]

# Optional: fixed output location (otherwise <CWD>/_testing).
[paths]
output_dir = "/path/to/_testing"
```

In the common case `engines.toml` carries only the engine. `sr_specs` and the
on-disk test packs come from `project_config.toml` (set `SIMREADY_PROJECT_CONFIG`
or pass `--project-config`). `engines.toml` can also carry `[paths].sr_specs`
and `[tests].paths`, but those are overrides for when there is no project config
or a machine needs a different value; resolution is first-non-empty-wins in the
order CLI flag, environment variable, `engines.toml`, then `project_config.toml`.

Verify with `simready-benchmark --show-config`. This prints the resolved engines.toml,
sr_specs source, project_config, and discoverable test packs with
`[ok]`/`[missing]` markers.

---

## CLI

`simready-benchmark` is a single command with mutually-exclusive **mode flags**
rather than subcommands. The default invocation runs the full pipeline.

### Mode Flags

| Flag | Behavior |
|---|---|
| (default) | Plan + run + report in one pass |
| `--plan-only` | Generate plan.json and stop |
| `--edit-plan ...` | Edit an existing plan.json (no run, no report) |
| `--resume` | Run + report against an existing plan.json |
| `--report-only` | Regenerate the report from existing per-test result.json files |
| `--list-tests` | Print the registered features and tests |
| `--show-config` | Print the resolved configuration (engines.toml + tier, sr_specs source, test packs discoverable) and exit |
| `--init-engines-toml [PATH]` | Run the setup wizard and write a complete `engines.toml`. Overwrites an existing file by default; pass `--no-overwrite` to refuse |

Plus the `stamp` subcommand:

```bash
simready-benchmark stamp --output-dir _testing
```

### Common Flags

| Flag | Description |
|---|---|
| `--project-config <PATH>` | Load content roots, `sr_specs`, profiles, features, and `[tests].paths` from project_config.toml |
| `--sr-specs <PATH>` | Path to sr_specs (overrides project config and engines.toml) |
| `--assets <PATH> [<PATH>...]` | One or more files or directories of assets |
| `--engines-toml <FILE>` | Override the engines.toml lookup chain |
| `--tests-path <PATH> [<PATH>...]` | Extra directories scanned for `@test`-decorated functions |
| `--output-dir <DIR>` | Override the default `_testing/` location |
| `--features <ID> [<ID>...]` | Match feature ID prefixes; also bypasses the validation gate for matches |
| `--tests <NAME> [<NAME>...]` | Exact test names to keep |
| `--max-concurrent <N>` | Up to N engine sessions in parallel (default 2) |
| `--workers <url1,url2,...>` | NVCF worker endpoints (additive to local Kit engines) |
| `--no-local-engines` | NVCF-only mode (use ONLY explicitly-constructed engines) |
| `--runtime <NAME> [<NAME>...]` | Pin execution to one or more named engines from engines.toml (for example `--runtime isaac_sim`); unknown names hard-fail |
| `--no-stamp` | Skip the stamp phase after a successful run |

### Exit Codes

| Code | Meaning |
|------|---------|
| `0` | All tests passed |
| `1` | One or more tests failed, or a runtime error |
| `2` | CLI usage error |
| `3` | `--resume` / `--report-only` invoked but no plan.json or results found |
| `4` | Setup or environment not ready (no engine could run part of the plan, or an engine failed preflight). `4` outranks `1`. |
| `130` | Interrupted by user (Ctrl-C) |

---

## Python API (Advanced)

The CLI is the recommended entry point. The Python surfaces are low-level
and intended for advanced integrations:

- `simready_benchmark.Planner`. Static-method class; `Planner.plan(sr_specs_path, test_dirs, discovery, validation_source, ...) -> Plan`.
- `simready_benchmark.Runner`. Static-method class; `Runner.start(plan, output_dir, ...) -> RunHandle`.
- `simready_benchmark.Reporter`. Static-method class; `Reporter.generate(output_dir, fmt="both") -> dict`.
- `simready_benchmark.PlanEditor`. Wraps a `Plan`; appends overrides. Load and save through `simready_benchmark.core.models.load_plan` / `save_plan`.
- `simready_benchmark.test`. The `@test` decorator. Functions MUST be `async def`. Required kwargs: `features`, `requirement`, `name`, `description`, `expected_video`, `version`.
- `simready_benchmark.RunContext`, `simready_benchmark.SceneHandle`. Runtime types used by tests.

Refer to the ten packaged skills under `src/simready_benchmark/skills/` for
ready-to-use recipes: `configure-engines-toml`, `run-tests`, `edit-plan`,
`stamp-results`, `embed-in-pipeline`, `diagnose-failures`, `remediate-assets`,
`write-test-pack`, `write-engine-plugin`, and `write-online-runner`.

---

## Metadata Ownership

`simready.test` owns only the `runtime_testing` sub-key of `SimReady_Metadata`
in USD `customLayerData`, and the `runtime_tests[]` array in
`.simready/validation.json`. It never reads or writes `asset_id`,
`validation`, or any other section.

```
customLayerData["SimReady_Metadata"]
├── asset_id          <- simready.create
├── validation        <- simready.validate
└── runtime_testing   <- simready.test
```

```
.simready/validation.json
├── schema_version, profile, passed, issues, ...   <- simready.validate
└── runtime_tests[]                                 <- simready.test
```

---

## License

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
