Metadata-Version: 2.5
Name: turbobench-cli
Version: 2.0.4
Summary: Correctness-gated, provider-neutral reinforcement-learning environment benchmarks
Project-URL: Homepage, https://github.com/tsilva/turbobench
Project-URL: Issues, https://github.com/tsilva/turbobench/issues
Project-URL: Repository, https://github.com/tsilva/turbobench
Author: Tiago Silva
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: numpy<3,>=1.26
Requires-Dist: packaging<27,>=24
Requires-Dist: pillow<13,>=10.4
Description-Content-Type: text/markdown

<div align="center">
  <img src="./logo.png" alt="turbobench" width="360" />

  **⚖️ Matched environments. Measured fairly. ⚖️**
</div>

`turbobench` is a local Python CLI for reinforcement-learning environment authors,
researchers, and provider maintainers who need fair performance comparisons between
compatible implementations and standardized cross-provider parity checks. Immutable
parity profiles pin the original authority, exact checks, full workload, and quick
development workload. Both parity and benchmark runs produce portable,
self-verifying evidence.

Optional comparison videos replay the same locked providers and semantic action trajectory.
Only valid, conclusive evidence can produce unmarked promotional media; diagnostic output is
clearly watermarked.

## Install

Install [turbobench-cli 2.0.0](https://pypi.org/project/turbobench-cli/2.0.0/)
from PyPI:

```bash
uv tool install turbobench-cli==2.0.0
```

Alternatively, install it in an active virtual environment with
`python -m pip install turbobench-cli==2.0.0`. The installed command and Python
import remain `turbobench`.

For a development checkout:

```bash
git clone https://github.com/tsilva/turbobench.git
cd turbobench
uv sync --frozen --group dev
```

Run `turbobench profiles list` and `turbobench providers list` to choose a
compatible profile and provider pair. Prefix CLI commands with `uv run` when
working from a development checkout.

## Commands

```bash
turbobench doctor vizdoom/basic-v1       # check the host, tools, and profile assets
turbobench profiles list                 # list immutable workloads
turbobench providers list                # list built-in and registered providers

turbobench parity supermario/canonical-v2 \
  --candidate env-supermariobrosnes-turbo-emu@checkout:/absolute/path/to/repo \
  --allow-dirty --quick                         # test current work diagnostically

turbobench parity vizdoom/basic-v2 \
  --candidate env-vizdoom-turbo@artifact:/absolute/path/to/final.whl \
  --output turbobench-parity/vizdoom            # certify the exact final wheel

turbobench verify-parity turbobench-parity/vizdoom --require-canonical

turbobench compare vizdoom/basic-v1 \
  --left env-vizdoom-turbo@1.3.0.post27 \
  --right vizdoom@1.3.0 \
  --output turbobench-results/vizdoom            # create a result bundle

turbobench compare breakout/start-v3 \
  --left env-breakoutatari2600-turbo-native@VERSION \
  --right stable-retro@1.0.1 \
  --output turbobench-results/breakout-vs-stable-retro

turbobench compare breakout/start-v3 \
  --left env-breakoutatari2600-turbo-native@VERSION \
  --right env-stableretro-turbo@VERSION \
  --output turbobench-results/breakout-vs-stable-retro-turbo

turbobench verify turbobench-results/vizdoom  # verify integrity and consistency
turbobench report turbobench-results/vizdoom  # print the generated report
turbobench promo turbobench-results/vizdoom --diagnostic

uv run --frozen ruff check .                    # lint the project
uv run --frozen pytest -m "not acceptance"      # run tests without proprietary assets
```

Long-running commands write progress to standard error and reserve standard output for their
final machine-readable JSON.

## Notes

- The controller supports Python 3.11 and newer. Provider runtimes default to CPython 3.14.
  `uv`, FFmpeg, and FFprobe are required.
- Current performance profiles cover `supermario/canonical-v1`,
  `breakout/start-v3`, and `vizdoom/basic-v1`. Historical profiles remain
  available for verifying their existing result bundles. Shapes 1, 16, and 32
  are measured and reported independently.
- Provider references accept `provider`, `provider@latest`, `provider@VERSION`,
  `provider@artifact:/absolute/path.whl`, and `provider@checkout:/absolute/path`.
  `latest` excludes prereleases, yanked releases,
  incompatible artifacts, and releases still inside the seven-day quarantine.
- Set `TURBOBENCH_ROM_PATH`, `TURBOBENCH_ASSET_ROOT`, or `RETRO_DATA_PATH` to locate required
  local game payloads. ROMs and local paths are never written to portable bundles; only
  canonical digests are recorded.
- The `breakout/start-v3` profile compares the Atari 2600 Breakout `Start`
  workload against either original Stable Retro or Stable Retro Turbo. Replace
  `VERSION` with an exact release, use `@latest`, or select a clean checkout
  with `@checkout:/absolute/path`.
- Every official result must pass provider compatibility, matched correctness, system-load,
  alternating paired-measurement, statistical uncertainty, provenance, and asset gates.
  Quick runs and explicit overrides remain diagnostic.
- Exact release parity accepts only the final local wheel on the chosen canonical host.
  Checkout snapshots include tracked edits and nonignored untracked source, so developers
  do not need to commit before running a quick diagnostic check.
- Turbo providers are preflighted against the normative
  [Turbo Vector API v2 contract](docs/TURBO_VECTOR_API_V2.md). Contract reports
  are hash-bound into result bundles; malformed v2 providers stop before any
  workload, while historical v1 providers remain diagnostic-only.
- Result bundles contain the exact provider lock, shape-local statistics, report, chart, raw
  evidence, verification records, and optional media. `manifest.json` binds every portable
  file by size and SHA-256; turbobench does not upload or publish bundles.
- Official v1 hosts are Apple-silicon macOS and x86-64 Linux. Third-party providers can
  register through the `turbobench.providers` entry-point group.

## Architecture

![turbobench architecture](./architecture.png)

## License

[MIT](LICENSE)
