Metadata-Version: 2.4
Name: benchmatrix
Version: 0.2.3
Summary: A pytest-benchmark matrix, metadata, and JSON results layer.
Project-URL: Homepage, https://github.com/ryancswallace/benchmatrix
Project-URL: Changelog, https://github.com/ryancswallace/benchmatrix/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/ryancswallace/benchmatrix#readme
Project-URL: Issues, https://github.com/ryancswallace/benchmatrix/issues
Project-URL: Source, https://github.com/ryancswallace/benchmatrix
Author-email: Ryan Wallace <ryancswallace@gmail.com>
Maintainer-email: Ryan Wallace <ryancswallace@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: benchmark,benchmarking,performance,pytest,pytest-benchmark
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Software Development :: Testing
Requires-Python: <3.15,>=3.11
Requires-Dist: pytest-benchmark>=5.2.3
Requires-Dist: pytest>=9.1.1
Description-Content-Type: text/markdown

# benchmatrix

[![CI](https://github.com/ryancswallace/benchmatrix/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/ryancswallace/benchmatrix/actions/workflows/ci.yml)
[![Documentation](https://github.com/ryancswallace/benchmatrix/actions/workflows/docs.yml/badge.svg?branch=main)](https://github.com/ryancswallace/benchmatrix/actions/workflows/docs.yml)
[![Docker](https://github.com/ryancswallace/benchmatrix/actions/workflows/docker.yml/badge.svg?branch=main)](https://github.com/ryancswallace/benchmatrix/actions/workflows/docker.yml)
[![CodeQL](https://github.com/ryancswallace/benchmatrix/actions/workflows/codeql.yml/badge.svg?branch=main)](https://github.com/ryancswallace/benchmatrix/actions/workflows/codeql.yml)
[![OpenSSF Scorecard](https://github.com/ryancswallace/benchmatrix/actions/workflows/scorecard.yml/badge.svg?branch=main)](https://github.com/ryancswallace/benchmatrix/actions/workflows/scorecard.yml)
[![Workflow lint](https://github.com/ryancswallace/benchmatrix/actions/workflows/workflow-lint.yml/badge.svg?branch=main)](https://github.com/ryancswallace/benchmatrix/actions/workflows/workflow-lint.yml)
[![Python 3.11-3.14](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-3776AB?logo=python&logoColor=white)](https://github.com/ryancswallace/benchmatrix/blob/main/pyproject.toml)
[![Typed with basedpyright](https://img.shields.io/badge/types-basedpyright-2f6fdd)](https://github.com/DetachHead/basedpyright)
[![Linted with Ruff](https://img.shields.io/badge/lint-Ruff-46a2f1)](https://docs.astral.sh/ruff/)
[![Coverage gate: 95%](https://img.shields.io/badge/coverage%20gate-%E2%89%A595%25-2e7d32)](https://github.com/ryancswallace/benchmatrix/blob/main/pyproject.toml)
[![SBOM: CycloneDX 1.6](https://img.shields.io/badge/SBOM-CycloneDX%201.6-6f42c1)](https://cyclonedx.org/)

**Benchmark matrices for Python projects that need performance data they can
trust, compare, and parse.**

benchmatrix sits on top of
[pytest-benchmark](https://pytest-benchmark.readthedocs.io/) and adds the layer
that benchmark suites usually grow by hand: implementation-by-case matrices,
strict JSON-safe metadata, metric-aware result parsing, and concise display of
saved benchmark runs.

| Build repeatable suites | Keep metrics honest | Parse saved runs |
| --- | --- | --- |
| Generate pytest benchmark tests across implementations, cases, and metric views. | Separate latency, throughput, and local distribution comparisons instead of mixing unlike numbers. | Load benchmatrix-tagged pytest-benchmark JSON rows into structured Python objects. |

## Quick Start

Create a benchmark matrix from ordinary synchronous callables:

```python
from benchmatrix import BenchmarkCase, make_benchmark_test

implementations = {
    "builtin": sum,
    "loop": lambda vs: sum(v for v in vs),
}

cases = [
    BenchmarkCase.from_values(
        "small",
        list(range(100)),
        work_units=100,
        work_unit_name="items",
    ),
]

test_sum_matrix = make_benchmark_test(implementations, cases)
```

Run it with pytest-benchmark and keep the machine-readable output:

```bash
uv run pytest tests/test_sum_benchmark.py --benchmark-json benchmark.json
```

Read the tagged rows back later:

```python
from benchmatrix import display_benchmark_rows, load_benchmark_json

rows = load_benchmark_json("benchmark.json")
display_benchmark_rows(rows)
```

## Why It Exists

pytest-benchmark owns timing, calibration, statistics, terminal reporting, and
JSON export. benchmatrix owns the repeatable structure around those timings.

| Need | benchmatrix gives you |
| --- | --- |
| Compare multiple implementations | One generated pytest benchmark matrix instead of repeated parametrization code. |
| Track what each timing means | JSON-safe invocation metadata with implementation, case, and metric identity. |
| Report different metric views | Single-call latency, logical-work throughput, and local tail-latency summaries. |
| Reuse benchmark output | Parsers and display helpers for pytest-benchmark JSON produced by benchmatrix tests. |

benchmatrix is intentionally narrow: it benchmarks synchronous Python callables.
It is not a load-testing framework, production latency monitor, or replacement
for pytest-benchmark.

## Install

For local development from this repository:

```bash
make ready
```

To try the package from another project before the first PyPI release, install
from GitHub:

```bash
uv add "benchmatrix @ git+https://github.com/ryancswallace/benchmatrix"
```

## Documentation

| Start here | Use it for |
| --- | --- |
| [First benchmark](docs/tutorials/first-benchmark.md) | A complete first benchmark from test file to parsed JSON. |
| [Create a benchmark matrix](docs/how-to/create-benchmark-matrix.md) | Cases, work units, fresh inputs, and synchronous target wrappers. |
| [Parse benchmark results](docs/how-to/parse-results.md) | Loading and displaying benchmatrix-tagged pytest-benchmark JSON. |
| [Performance model](docs/explanation/performance.md) | What the metrics mean and what they do not prove. |
| [Configuration and automation](docs/reference/configuration.md) | Make targets, CI workflows, Docker checks, docs, and SBOM generation. |

The MkDocs site builds in strict mode and generates API reference pages from the
package docstrings.

## Project Links

* [Examples](examples/)
* [Contributing](CONTRIBUTING.md)
* [Changelog](CHANGELOG.md)
* [Security policy](SECURITY.md)
* [Release policy](RELEASING.md)
* [Code of conduct](CODE_OF_CONDUCT.md)
* [Citation metadata](CITATION.cff)

## License

benchmatrix is distributed under the [MIT License](LICENSE).
