Metadata-Version: 2.4
Name: simready-evaluate
Version: 2026.10.0
Summary: SimReady Evaluate, the engine-agnostic core for validating SimReady USD assets
Author: NVIDIA Corporation
License-Expression: Apache-2.0
Project-URL: Homepage, https://www.nvidia.com
Keywords: nvidia,simready,test,validation
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: filelock<4.0,>=3.13.0
Requires-Dist: usd-core>=23.5
Provides-Extra: kit
Requires-Dist: simready.evaluate.engine-kit<2026.11,>=2026.9.0rc3; extra == "kit"
Provides-Extra: s3
Requires-Dist: boto3>=1.42.58; extra == "s3"
Requires-Dist: urllib3<3,>=2.8.0; extra == "s3"
Provides-Extra: search
Requires-Dist: simready-search>=2026.10.0-rc0; extra == "search"
Dynamic: license-file

# simready-evaluate

Engine-agnostic Python framework for evaluating SimReady USD assets.
Discovers applicable tests, executes them across one or more engine sessions
in parallel, and produces HTML and JSON reports.

Part of the SimReady Python Library Suite alongside `simready-validate` and
`simready-search`.

Evaluate replaces Benchmark execution starting with the 2026.10 release. If
you are upgrading from `simready-benchmark` 2026.8, follow the
[migration guide](https://gitlab-master.nvidia.com/omniverse/kit-extensions/simready-explorer/-/blob/release-2026.10.0/source/libraries/evaluate/docs/migration-from-benchmark.md)
before installing. Old Benchmark engine plugins can conflict with Evaluate's
`kit` registration even when `pip check` succeeds.

---

## Requirements

- **Python 3.12 exactly.** The supported engine, NVIDIA Isaac Sim, requires
  Python 3.12, so the virtual environment must use 3.12.
- **A 64-bit x86 machine (x86_64, also called AMD64)** running Windows 10 or 11,
  or Linux (Ubuntu). `usd-core`, a core dependency, publishes no ARM (aarch64)
  wheels, so `pip install` fails while resolving `usd-core` on ARM machines,
  including ARM cloud instances. On Linux, confirm your architecture with
  `uname -m`, which must report `x86_64`.
- **A CUDA-capable NVIDIA GPU and about 30 GB of free disk**, primarily for
  [Isaac Sim](https://docs.isaacsim.omniverse.nvidia.com/latest/index.html),
  the engine that runs the tests.

---

## Installation

Always work inside a dedicated virtual environment, because the framework and
the Isaac packages can conflict with other USD or Omniverse installations.

### Windows

```powershell
mkdir C:\simready
cd C:\simready
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install "simready-evaluate[kit]"
```

If PowerShell reports that running scripts is disabled, run
`Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser` one time,
then retry the activate command.

### Linux (Ubuntu)

Ubuntu 22.04 ships Python 3.10 and has no `python3.12` package, so add the
[deadsnakes PPA](https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa) first:

```bash
sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update
sudo apt install -y python3.12 python3.12-venv
```

Then create the environment and install:

```bash
mkdir -p ~/simready
cd ~/simready
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install "simready-evaluate[kit]"
```

Do not install `python-is-python3` as a substitute; it only aliases `python` to
the system Python (3.10 on Ubuntu 22.04), which Isaac Sim rejects.

### Verify

```bash
simready-evaluate --version
```

These commands respect the default index, mirror, and credentials already
configured for `pip`. To install a local repository build, run
`repo.bat python_package` from the `simready-explorer` root and install the
`simready_evaluate-*.whl` and
`simready_evaluate_engine_kit-*.whl` files from `_build/packages/dist/`.

The current packaging tool clears the complete generated staging/dist
directories even with `--package` filters. A filtered build regenerates only
selected wheels, not the other local wheel files. Use the unfiltered command
above to regenerate the full configured wheel set, or an explicitly isolated
build workspace/output location for a targeted build. This does not uninstall
active environments or edit source assets.

The `[kit]` extra pulls in `simready.evaluate.engine-kit`, the Kit and Isaac Sim
engine plugin, along with `usd-core`, `numpy`, and `Pillow`. Install
`simready-evaluate` without the extra for the engine-agnostic core and CLI only.

Specification tiers may also be installed as Python packages:

```bash
pip install simready-foundation-tier-core
```

Evaluate discovers their catalogs and optional runtime tests through the
`simready.tier` descriptor. Independent test-only wheels use
`simready.evaluate.tests`.
An explicit Foundation checkout remains available for source-tree development:

```bash
simready-evaluate --foundations-path /path/to/simready_foundations --list-tests
```

This discovers the checkout's tier catalogs and runtime-test package. The flag
applies to that command only; it is not required for an installed-tier
workflow.

Foundations 2 readers accept neutral `FET` and `CAP` IDs as `<name>.<version>` and
runtime variants as `<name>.<runtime>.<version>`. Earlier explicit
`.OpenUSD.<version>` IDs normalize to the neutral spelling in catalogs and
validation receipts. Versions remain opaque tokens. Profile selectors support
family, runtime, and version wildcards. Bare profile selectors are invalid:
`FET004_Multibody.*` selects all versions and runtimes, and `FET004_Multibody.0*`
selects neutral versions starting with `0`. Use `FET004_Multibody.*.0*` for those
versions across every runtime, `FET004_Multibody.PhysX.*` for every PhysX version, or
`FET004_Multibody.OpenUSD.0*` for neutral `0` versions. Capability selectors use
the same policy. Existing profiles and upstream resolvers must align with this
policy. `name.OpenUSD.*` is rejected in profiles and CLI normalization because
removing `OpenUSD` would produce `name.*`, broadening scope to every runtime.
Exact IDs and dependencies retain their neutral-ID meaning.
CLI feature prefixes accept both neutral spellings and retain their existing
behavior independently of profile selectors.

To use the same checkout on every command without repeating
`--foundations-path`, reuse the per-user settings file listed under
[Engine Configuration](#engine-configuration), or select a separate file with
`--engines-toml FILE`, and persist the checkout's project configuration:

```toml
[paths]
project_config = "<foundations>/sample_content/project_config.toml"
```

Use forward slashes in the TOML path on Windows. Evaluate then obtains the
checkout's content root, tier catalogs, and tier-owned runtime tests from that
single project configuration, so subsequent commands need neither
`--foundations-path` nor `--project-config`:

```bash
simready-evaluate --show-config
simready-evaluate --list-tests
simready-evaluate --assets /path/to/assets
```

`SIMREADY_PROJECT_CONFIG` provides the same project-config selection through an
environment variable. Explicit CLI arguments take precedence over environment
and `engines.toml` defaults.

---

## Setup

Install a tier with its Evaluate extra, then validate the environment:

```bash
pip install "simready-foundation-tier-core[evaluate]"
```

The tier advertises catalogs and its optional `runtime_tests_path` through
`simready.tier`. Independent test-only distributions use
`simready.evaluate.tests`. Setup checks those installed providers and the active
Python 3.12 environment. Do not install pip Isaac beside Evaluate's `usd-core`:
Isaac's converter dependencies bring `usd-exchange`, and the two distributions
install overlapping `pxr` files. `--setup --install-isaac` refuses that unsafe
combination so it cannot silently break `usd-validation-nvidia`.

Create a separate Python 3.12 venv containing only pip Isaac, then point
`engines.toml` at its `isaacsim` launcher and Python executable:

```toml
[kit.isaac_sim]
executable_path = "/path/to/isaac-env/bin/isaacsim"
python_executable = "/path/to/isaac-env/bin/python"
version = "6.2.0.0"
tags = ["isaac", "kit"]
runtime = "PhysX"
```

Add a second stanza with `runtime = "Newton"` and
`experience = "isaacsim.exp.full.newton"` for Newton. On Windows, use the
environment's `Scripts/isaacsim.exe` and `Scripts/python.exe`. See the
installation guide's **Separate Isaac environment** section for complete
commands and configuration.

Setup creates no source checkout, does not persist `--foundations-path`, and
does not create a configuration file. A source-checkout-only workflow does not
use `--setup` to register the checkout; select it per command with
`--foundations-path`, or persist its `sample_content/project_config.toml` under
`[paths]` in `engines.toml` as shown above.

For the recommended separate-Isaac layout, verify the host with `python -m pip
check`, `simready-evaluate --show-config`, `--list-tests`, and a targeted real
session. `--setup --engines-toml engines.toml --no-install-isaac` checks a
prepared external runtime read-only. For Script Editor execution, use the
[isolated installer](https://gitlab-master.nvidia.com/omniverse/kit-extensions/simready-explorer/-/blob/release-2026.10.0/source/libraries/evaluate/docs/install.md#prepare-isaac-for-script-editor-execution)
and native canary; its precise bundled-SDK dependency qualifications do not
mean that runtime `pip check` was clean.

Use a fresh Python 3.12 environment for Evaluate. Core's current runtime tests
live in `simready.foundation.tier_core.runtime_tests`; former flat names remain
provider aliases, while Python code uses the namespaced imports. See
[Compatibility](https://gitlab-master.nvidia.com/omniverse/kit-extensions/simready-explorer/-/blob/release-2026.10.0/source/libraries/evaluate/docs/compatibility.md)
and the [installation guide](https://gitlab-master.nvidia.com/omniverse/kit-extensions/simready-explorer/-/blob/release-2026.10.0/source/libraries/evaluate/docs/install.md).
Use a fresh environment, or explicitly uninstall `simready-benchmark`,
`simready-benchmark-engine-kit`, `simready-benchmark-kit-suite`, and
`simready-foundation-runtime-tests-kit` before installing Evaluate into an
existing environment. The setup command detects retained retired
distributions and asks for remediation.

Pip Isaac in the active environment can be discovered automatically as two
runtimes: `isaac_sim` for PhysX and `isaac_sim_newton` for Newton. Keep pip
Isaac separate from Evaluate's `usd-core` environment to avoid overlapping
USD files, and use `engines.toml` for that recommended layout, a custom
executable, custom experience, or remote worker.

### Verify the Configuration

```bash
simready-evaluate --show-config
```

This prints installed tier catalogs, runtime-test packs, automatic engines, and
any explicit project or `engines.toml` override.

---

## First Test

List the discovered tests, then run the full pipeline (plan, run, and report)
against one demo asset. With an installed tier, or after persisting the
Foundation project config in `engines.toml`, run:

```bash
simready-evaluate --list-tests
simready-evaluate --assets <foundations>/sample_content/common_assets/props_general/apple_a01
```

For an unconfigured source checkout, include `--foundations-path` on both
commands:

```bash
simready-evaluate --foundations-path <foundations> --list-tests
simready-evaluate --foundations-path <foundations> \
  --assets <foundations>/sample_content/common_assets/props_general/apple_a01
```

Replace `<foundations>` with your clone path, for example
`<drive>:\path\to\simready_foundations` on Windows or
`/path/to/simready_foundations` on Linux. Isaac Sim launches and the
applicable tests run against the apple asset. Evaluate prints the allocated
run directory and the exact HTML report path when it finishes. Open that
printed `index.html`; a missing or empty requested directory is used directly,
while a nonempty directory receives a fresh numbered `evaluate_report_N` child.

Exit code `0` means no failed runtime work or unavailable required runtime
bindings; it does not certify that every test or feature passed. Failed tests
and unavailable required bindings produce `1`, while engine-readiness failures
produce `4`. Refer to [Exit Codes](#exit-codes) for the full list.

### Known Issues (Isaac Sim First Run)

- **The first run on a machine where Isaac Sim has never launched can be very
  slow or appear to fail** until Isaac Sim compiles its shader cache. Warm the
  cache once: launch Isaac Sim directly, wait for the main viewport, open any USD
  file (the sample apple works), then close it. Re-run the evaluation. After one
  successful run the cache is warm and later runs complete normally.
- **A test can fail with an entirely white (blank) comparison frame.** Isaac Sim
  and Kit sometimes do not render a frame in time, which fails the comparison.
  This is an engine rendering issue, not an asset defect. Re-run only the failed
  tests with `simready-evaluate --rerun failed --output-dir <resolved-run-dir>`.
  A passing retry establishes only the retested coverage, not full asset
  certification. Use the resolved output directory emitted by the original
  run (which may differ from the requested directory), and retain the same
  `--project-config` and effective engine/test configuration.
- **The separate Isaac Sim install stalls or fails on a network read timeout.**
  Re-run the pip install with `--timeout 120 --retries 5` inside the Isaac venv.
  Do not install it into the validation venv as a retry.
- **An installed provider or Isaac is missing.** Run
  `simready-evaluate --show-config`. If Isaac is missing, verify the separate
  venv's `executable_path` and `python_executable` entries in `engines.toml`.

---

## Day-to-Day Use

The virtual environment is per-terminal. In every new terminal, activate it
before running the tool:

```bash
# Windows
cd C:\simready
.\.venv\Scripts\activate

# Linux
cd ~/simready
source .venv/bin/activate
```

Tier-owned test content and the framework update through pip in the Evaluate
venv, not through `--update`:

```bash
python -m pip install --upgrade "simready-foundation-tier-core[evaluate]" "simready-evaluate[kit]"
simready-evaluate --update --no-update-isaac
```

The second command reports the installed framework versions; it does not
upgrade packages. Update the separate Isaac venv independently as described in
the [installation guide](https://gitlab-master.nvidia.com/omniverse/kit-extensions/simready-explorer/-/blob/release-2026.10.0/source/libraries/evaluate/docs/install.md). A plain `--update` checks only
for pip Isaac in the *active* venv, so it cannot maintain the recommended
separate Isaac installation.

---

## Output Layout

The requested output location resolves in this order: `--output-dir`,
`engines.toml [paths].output_dir`, `<project_root>/_testing`, then
`<cwd>/_testing`. A missing or empty requested directory is used directly.
Only a nonempty directory receives an `evaluate_report_N` child, starting at `0`
or one above the highest existing numeric suffix. The resolved directory contains:

- `plan.json`. The resolved plan
- `run_summary.json`. Terminal run, readiness, and failure summary
- `state/events.jsonl`. Live JSON-Lines event stream
- `state/work_pool.json`. Work-item and per-test execution state
- `results/<asset-dir>/<asset>`. Root-layer report copy; writable layers are
  stamped, while USDZ packages remain byte-identical
- `results/<asset-dir>/.simready/assets/<asset-key>/validation.json`. Wrapped
  `SimReady_Metadata.runtime_testing` receipt
- `results/<asset-dir>/.simready/assets/<asset-key>/runtime/<test-output-name>/<engine>/result.json`.
  Unique tests keep their raw name; provider qualification and a deterministic
  digest make overlapping names collision-safe. Per-engine
  test result and media artifacts
- `logs/<engine>_<n>.log`. Per-engine subprocess logs
- `index.html` plus `assets/<slug>.html` (HTML index and per-asset details) and
  `report/test_results_index.json`. Aggregated reports

The original asset directory is never modified. This is intentional: source
assets may be read-only, remote, version controlled, or require a separate
checkout/publish workflow before they can be changed.

The asset namespace key is the first 16 hexadecimal characters of SHA-256 of
the complete UTF-8 USD filename, including its extension. Sibling USD files
therefore keep separate receipts and runtime evidence without changing their
mirrored relative paths. Readers support old shared receipt/result paths as
read-only compatibility inputs; shared results without an exact asset identity
cannot certify coverage.

The results tree mirrors the source asset's relative path to prevent output
name collisions; it does not mirror the complete asset directory. Stamping
copies only the root USD layer and writes the evaluation receipt beside it.
Referenced layers, payloads, `materials/`, `textures/`, and other relative
dependencies are not copied or rewritten. Consequently, the stamped USD under
`results/` is a report/metadata artifact and may not open with complete
composition or visual fidelity. Open the original asset for inspection, or use
an authorized downstream packaging/publishing step if a portable stamped asset
is required.

Evaluate never clears the requested output directory. Missing or empty directories
are used directly; nonempty directories receive an `evaluate_report_N` child,
starting at `0` or one above the highest existing numeric suffix. Gaps are not
reused, and existing files are preserved. Reruns and report regeneration use
the exact original run directory without allocating a new one.
Read the resolved output and report paths from
the final CLI events. The deprecated `--keep-outputs` flag is accepted for
compatibility but does not change the always-preserve behavior.

---

## Key Concepts

### Plan / Run / Stamp / Report Pipeline

A single `simready-evaluate` invocation runs four phases by default:

1. **Plan.** Scan installed test packs and the active `sr_specs`; match assets to
   applicable tests; write `plan.json`.
2. **Run.** Execute the plan across one or more engine sessions in parallel;
   stream JSON events; write per-test `result.json`.
3. **Stamp.** Unless suppressed with `--no-stamp`, write matching evaluation
   metadata into writable root-layer mirrors and their asset-owned
   `.simready/assets/<asset-key>/validation.json` receipts.
   USDZ mirrors are byte-identical and sidecar-only. These copies are receipts,
   not self-contained asset packages.
4. **Report.** Aggregate per-test results and stamp information into HTML and
   JSON reports.

### Engines and Test Packs

Engines are registered through `simready.evaluate.engines`. Independent test
packs use `simready.evaluate.tests`; tier-owned packs are advertised through a
`simready.tier` descriptor's optional `runtime_tests_path`. No per-install path
configuration is needed for installed providers.

Out of the box, `simready-evaluate` registers an experimental `nvcf` engine
plugin for NVIDIA Cloud Functions execution. The NVCF plugin, remote-worker
path, and local emulator are **Technology Preview** features and are not the
supported release-validation path. They require a separately provisioned
compatible worker endpoint and credentials; Evaluate does not provide an NVCF
deployment, worker URL, API key, or test account.
`simready.evaluate.engine-kit` adds the supported `kit` engine plugin for Kit
and Isaac Sim. Third-party engines and test packs slot in through the same
mechanism.

### Engine Configuration

Pip Isaac in the active environment can be discovered automatically as
`isaac_sim` (PhysX) and `isaac_sim_newton` (Newton). Keep it in a separate
environment from Evaluate's `usd-core` for the supported isolated layout;
configure that environment explicitly below.

An optional `engines.toml` overrides automatic local discovery for a
standalone/custom runtime. The framework reads it using this lookup order:

1. `--engines-toml <FILE>` CLI flag
2. `$SIMREADY_ENGINES_TOML` environment variable (a missing file is a hard error)
3. `engines.toml` in the current working directory
4. `~/.simready-benchmark/engines.toml` or `~/.simready-evaluate/engines.toml`

On Windows, `~` is `%USERPROFILE%`. If both home files apply, their normalized
settings must agree. Use `--engines-toml FILE` or `SIMREADY_ENGINES_TOML` to
select explicitly when they differ. Reuse an existing home file rather than
creating a conflicting second configuration.

The run output uses its own precedence: `--output-dir`, then
`[paths].output_dir` from the resolved `engines.toml`, then
`<project_root>/_testing`, and finally `<cwd>/_testing`. Evaluate allocates a
fresh `evaluate_report_N` child only when the requested location is nonempty;
missing or empty directories are used directly.

For the recommended separate pip-Isaac environment, two engine
configurations can share its launcher and Python interpreter while selecting
different backends and Kit experiences:

```toml
[kit.isaac_sim]
executable_path = "C:/simready/isaac-env/Scripts/isaacsim.exe"
python_executable = "C:/simready/isaac-env/Scripts/python.exe"
version = "6.2.0.0"  # replace with the installed Isaac version
tags = ["isaac", "kit"]
runtime = "PhysX"

[kit.isaac_sim_newton]
executable_path = "C:/simready/isaac-env/Scripts/isaacsim.exe"
python_executable = "C:/simready/isaac-env/Scripts/python.exe"
version = "6.2.0.0"
tags = ["isaac", "kit"]
runtime = "Newton"
experience = "isaacsim.exp.full.newton"

# Optional: requested output location (otherwise project-root or CWD _testing).
[paths]
output_dir = "/path/to/evaluate_testing"
```

On Linux use the Isaac environment's `bin/isaacsim` and `bin/python`.
For a standalone installation, use its actual `.bat`, `.sh`, or native
launcher instead; omit `python_executable` unless that launcher has a
corresponding external Python interpreter.

Select an engine per run with `--runtime`: `--runtime isaac_sim_newton` runs
Newton only, `--runtime isaac_sim` runs PhysX only, and a run with no `--runtime`
runs every eligible test on all configured engines. Runtime-specific physics
tests are eligible only when the asset validation record contains the matching
feature variant, such as `FET_003_NEWTON` or `FET_003_PHYSX`;
`FET_003_STANDARD` alone does not authorize either concrete backend when the
test has runtime-specific variants. Runtime-neutral tests can still run on both
engines. The report attributes each result to the feature matching its engine.

To launch the raw Isaac Sim application in Newton mode, without the evaluation,
pass the experience as the first launcher argument:
`isaacsim.EXE isaacsim.exp.full.newton` (`isaacsim` without the `.EXE` on Linux).

---

## CLI

This README summarizes common operations. The project documentation's
`docs/cli-reference.md` contains every public option and subcommand. The exact
help for an installed build is available with `simready-evaluate --help` and
`simready-evaluate <subcommand> --help`.

The default invocation runs the full pipeline. Mutually exclusive mode flags
select alternate top-level workflows, and four subcommands provide staging,
stamping, and preview-emulator operations.

### Mode Flags

| Flag | Behavior |
|---|---|
| (default) | Plan, run, and report in one pass |
| `--plan-only` | Generate `plan.json` and stop |
| `--edit-plan ...` | Edit an existing `plan.json` (no run, no report) |
| `--rerun {plan,failed,report}` | Operate on an existing run: `plan` runs and reports against the existing `plan.json`, `failed` re-runs only the tests that failed in a previous run, `report` regenerates the report from existing per-test `result.json` files |
| `--list-tests` | Print the registered features and tests |
| `--show-config` | Print the resolved configuration and exit |
| `--setup` | Verify host providers and a configured prepared pip-Isaac runtime read-only; without configuration, retain legacy active-environment checks |
| `--update` | Check pip Isaac in the active environment for an update and report the Evaluate upgrade command; does not update a separate Isaac venv |

Subcommands are `stamp`, `stamp-one`, `stage`, and the Technology Preview
`nvcf-emulator`. For example:

```bash
simready-evaluate stamp --output-dir evaluate_testing
```

### Common Flags

| Flag | Description |
|---|---|
| `--project-config <PATH>` | Load content roots, `sr_specs`, profiles, features, and `[tests].paths` from `project_config.toml` |
| `--foundations-path <DIR>` | Load tier catalogs and runtime tests from a `simready_foundations` source checkout; cannot be combined with `--project-config` |
| `--sr-specs <PATH>` | Path to `sr_specs` (overrides project config and `engines.toml`) |
| `--assets <PATH> [<PATH>...]` | One or more files or directories of assets |
| `--engines-toml <FILE>` | Override the `engines.toml` lookup chain |
| `--tests-path <PATH> [<PATH>...]` | Extra directories scanned for `@test`-decorated functions |
| `--output <FILE>` | Override the plan path; normally unnecessary because `--output-dir` writes `<DIR>/plan.json` automatically |
| `--output-dir <DIR>` | Override the default `_testing/` requested location |
| `--features <ID> [<ID>...]` | Match feature ID prefixes; also bypasses the validation gate for matches |
| `--force-profile <NAME[@VERSION]>` | On a fresh run, diagnose one catalog profile without validation eligibility; labels the bypass and disables automatic asset stamping. Use an exact version when the name is ambiguous. See [CLI reference](https://gitlab-master.nvidia.com/omniverse/kit-extensions/simready-explorer/-/blob/release-2026.10.0/source/libraries/evaluate/docs/cli-reference.md#filters). |
| `--tests <SELECTOR> [<SELECTOR>...]` | Raw names keep every provider match; `provider:name` selects one exact provider |
| `--max-concurrent <N>` | Up to N engine sessions in parallel (default 2) |
| `--session-mode {batch,per_asset}` | Reuse sessions across assets, or start a fresh session for each asset |
| `--workers <url1,url2,...>` | **Technology Preview:** separately provisioned NVCF worker endpoints (additive to local Kit engines) |
| `--no-local-engines` | With `--workers`, skip local engines and select the **Technology Preview** NVCF-only path; without workers, local discovery is unchanged |
| `--runtime <NAME> [<NAME>...]` | Pin execution to one or more discovered engines (for example `isaac_sim_newton` or `isaac_sim`); with no flag every test runs on all matching engines. Unknown names hard-fail |
| `--no-stamp` | Disable post-run mirror receipt stamping |
| `--keep-outputs` | Deprecated compatibility flag; existing output is always preserved |
| `--format {html,json,both}` | Select report output; default `both` |

`--setup` accepts `--install-isaac` or `--no-install-isaac` for legacy
active-environment installations. Installation is refused beside `usd-core`.
Run `simready-evaluate --help` for the complete list.

### Exit Codes

| Code | Meaning |
|------|---------|
| `0` | No failed runtime work or unavailable required runtime bindings; skipped tests or static-validation failures may still be present |
| `1` | One or more tests failed, required runtime bindings were unavailable, a runtime error occurred, or requested receipt writing failed |
| `2` | CLI usage error |
| `3` | A requested existing plan or result set was missing (`--rerun plan`, `--rerun report`, or `--rerun failed`) |
| `4` | Setup or environment not ready (no engine could run part of the plan, or an engine failed preflight). `4` outranks `1`. |
| `130` | Interrupted by user (Ctrl-C) |

Exit `0` is not full asset certification: inspect the selected coverage and
static-validation results as well as runtime verdicts.

The HTML partitions discovered assets into **PASS / FAIL / NO VERDICT / Not tested**
in every engine view, with assessment reasons and separate test counts. Grey
Not tested identifies unselected runtime scope, not unavailable selected work.
Genuine N/A needs a structured applicability record; an unknown
historical skip or missing required assessment cannot establish PASS. A real
FAIL remains FAIL with an independent coverage warning. Required recorded
static failures can also establish FAIL; that is not a fresh validation run.
See [Reading Reports](https://gitlab-master.nvidia.com/omniverse/kit-extensions/simready-explorer/-/blob/release-2026.10.0/source/libraries/evaluate/docs/reading-reports.md) for the distinction between
these human-facing outcomes and unchanged persisted execution statuses.

Execution exceptions are incomplete coverage, not asset FAILs. Report and
receipt cells retain `execution_status: error` and `failure_category` alongside
canonical `status: incomplete`; legacy `passed: false` alone is not an asset
verdict. Automatic stamping finishes before `run_complete`. Its status/counts
are in `run_summary.stamping`; requested write failures set an otherwise
completed run to `artifact_failed`, without changing its test verdicts.
Per-asset stamp attempt identities keep current writes separate from historical
receipts retained after partial reruns. See the reading-reports guide.

---

## Python API (Advanced)

The CLI is the recommended entry point. The Python surfaces are low-level and
intended for advanced integrations:

- `simready.evaluate.Planner`. Static-method class;
  `Planner.plan(sr_specs_path, test_dirs, discovery, validation_source, ...) -> Plan`.
- `simready.evaluate.Runner`. Static-method class;
  `Runner.start(plan, output_dir, ...) -> RunHandle`.
- `simready.evaluate.Reporter`. Static-method class;
  `Reporter.generate(output_dir, fmt="both") -> dict`.
- `simready.evaluate.PlanEditor`. Wraps a `Plan`; appends overrides. Load and
  save through the public `Planner.load` and `Planner.save` methods.
- `simready.evaluate.test`. The `@test` decorator. Functions MUST be
  `async def`. Required keyword arguments: `features`, `name`, `description`,
  `expected_video`, and `version`. Optional: `engine`, `config_defaults`,
  `max_duration` (default 300), and `enabled` (default `True`).
- `simready.evaluate.RunContext`, `simready.evaluate.SceneHandle`. Runtime
  types used by tests.
- `simready.evaluate.EngineConfig`, `EnginePlugin`, `EngineSession`, and
  `ValidationResult`. Public engine-plugin contracts.

The package ships seven skills with
ready-to-use recipes: `install-environment`, `run-tests`,
`embed-in-pipeline`, `diagnose-failures`, `write-test-pack`, and
`write-engine-plugin`, plus `asset-repair-report` for authorized repair histories.
Locate their actual installed directory with `simready-evaluate --skills-path`.
The repair
report uses the public read-only `load_run_evidence(output_dir)` API to collect
canonical outcomes and native evidence without executing tests, stamping assets
or regenerating the original report. Static outcomes and missing runtime
coverage stay separate; a modified control's pass does not certify the original
asset. Its optional USD helpers require separately authorized application and
backups. See the installed skill's `references/tools.md` for script commands.

---

## Metadata Ownership

`simready.evaluate` owns only the `runtime_testing` sub-key of `SimReady_Metadata`
in USD `customLayerData` and `.simready/assets/<asset-key>/validation.json`.
It preserves `asset_id`, `validation`, and other sections rather than claiming
ownership of them. Legacy validation fields and unknown receipt keys are also
preserved when seeding the asset-owned receipt.

```
customLayerData["SimReady_Metadata"]
├── asset_id          <- simready.create
├── validation        <- simready.validate
└── runtime_testing   <- simready.evaluate
```

```
.simready/assets/<asset-key>/validation.json
└── SimReady_Metadata
    ├── validation                             <- simready.validate
    └── runtime_testing.tested_features         <- simready.evaluate
```

---

## License

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
