Metadata-Version: 2.4
Name: turbobench-cli
Version: 1.0.0
Summary: Correctness-gated, provider-neutral reinforcement-learning environment benchmarks
Project-URL: Homepage, https://github.com/tsilva/turbobench
Project-URL: Issues, https://github.com/tsilva/turbobench/issues
Project-URL: Repository, https://github.com/tsilva/turbobench
Author: Tiago Silva
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: numpy<3,>=1.26
Requires-Dist: packaging<27,>=24
Requires-Dist: pillow<13,>=10.4
Description-Content-Type: text/markdown

<div align="center">
  <img src="./logo.png" alt="turbobench" width="360" />

  **⚖️ Matched environments. Measured fairly. ⚖️**
</div>

`turbobench` is a local Python CLI for reinforcement-learning environment authors,
researchers, and provider maintainers who need fair performance comparisons between
compatible implementations. It verifies that two providers produce matching transitions
before timing them, then writes a portable, self-verifying evidence bundle. Run it with a
built-in workload profile and two provider references.

Optional comparison videos replay the same locked providers and semantic action trajectory.
Only valid, conclusive evidence can produce unmarked promotional media; diagnostic output is
clearly watermarked.

## Install

Install the published CLI from PyPI with `uv tool install turbobench-cli` or
`pip install turbobench-cli`. The installed command and Python import remain
`turbobench`.

For a development checkout:

```bash
git clone https://github.com/tsilva/turbobench.git
cd turbobench
uv sync --frozen --group dev
```

Run `uv run turbobench profiles list` and `uv run turbobench providers list` to choose a
compatible profile and provider pair.

## Commands

```bash
uv run turbobench doctor vizdoom/basic-v1       # check the host, tools, and profile assets
uv run turbobench profiles list                 # list immutable workloads
uv run turbobench providers list                # list built-in and registered providers

uv run turbobench compare vizdoom/basic-v1 \
  --left vizdoom-turbo@1.3.0.post23 \
  --right vizdoom@1.3.0 \
  --output turbobench-results/vizdoom            # create a result bundle

uv run turbobench verify turbobench-results/vizdoom  # verify integrity and consistency
uv run turbobench report turbobench-results/vizdoom  # print the generated report
uv run turbobench promo turbobench-results/vizdoom --diagnostic

uv run --frozen ruff check .                    # lint the project
uv run --frozen pytest -m "not acceptance"      # run tests without proprietary assets
```

Long-running commands write progress to standard error and reserve standard output for their
final machine-readable JSON.

## Notes

- The controller supports Python 3.11 and newer. Provider runtimes default to CPython 3.14.
  `uv`, FFmpeg, and FFprobe are required.
- Built-in profiles cover `supermario/canonical-v1`, `breakout/start-v1`, and
  `vizdoom/basic-v1`. Shapes 1, 16, and 32 are measured and reported independently.
- Provider references accept `provider`, `provider@latest`, `provider@VERSION`, and
  `provider@checkout:/absolute/path`. `latest` excludes prereleases, yanked releases,
  incompatible artifacts, and releases still inside the seven-day quarantine.
- Set `TURBOBENCH_ROM_PATH`, `TURBOBENCH_ASSET_ROOT`, or `RETRO_DATA_PATH` to locate required
  local game payloads. ROMs and local paths are never written to portable bundles; only
  canonical digests are recorded.
- Every official result must pass provider compatibility, matched correctness, system-load,
  alternating paired-measurement, statistical uncertainty, provenance, and asset gates.
  Quick runs and explicit overrides remain diagnostic.
- Result bundles contain the exact provider lock, shape-local statistics, report, chart, raw
  evidence, verification records, and optional media. `manifest.json` binds every portable
  file by size and SHA-256; turbobench does not upload or publish bundles.
- Official v1 hosts are Apple-silicon macOS and x86-64 Linux. Third-party providers can
  register through the `turbobench.providers` entry-point group.

## Architecture

![turbobench architecture](./architecture.png)

## License

[MIT](LICENSE)
