Metadata-Version: 2.4
Name: perfreport
Version: 0.0.2
Summary: Performance benchmarking framework with subprocess-isolated workers, text protocol, and HTML/Markdown reports with interactive SVG charts.
Author-email: mor <whiteluna@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/ssw-tech/perfreport
Project-URL: Issues, https://github.com/ssw-tech/perfreport/issues
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Operating System :: POSIX
Classifier: Operating System :: MacOS
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: psutil
Requires-Dist: pygal
Requires-Dist: cmarkgfm
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: uv; extra == "dev"
Dynamic: license-file

# perfreport

Python performance benchmarking framework. Runs workers as subprocesses, collects metrics via a lightweight text protocol, and renders HTML/Markdown reports with interactive SVG charts.

[![PyPI](https://img.shields.io/pypi/v/perfreport.svg)](https://pypi.org/project/perfreport/)
[![Python](https://img.shields.io/pypi/pyversions/perfreport.svg)](https://pypi.org/project/perfreport/)
[![License](https://img.shields.io/github/license/ssw-tech/perfreport.svg)](LICENSE)

## Features

- **Language-agnostic workers** — a worker is any process that writes protocol lines to stdout. Python, C, Go, bash, whatever.
- **Subprocess-isolated** — each scenario runs as its own process; the runner collects stdout, parses the protocol, aggregates results.
- **Hierarchical reports** — nested tree of test scenarios with cascading defaults.
- **Interactive SVG charts** — pygal-based line/bar charts embedded as `<object>` in HTML.
- **Matrix testing** — run the same benchmark against multiple Python versions and library versions (uv-based provisioning).
- **Configurable metrics** — built-in checkpoints/summary + custom per-driver metrics.
- **Multiple output modes** — interactive (TUI with prompts), verbose (streaming stdout), silent (file-only).

## Install

```bash
pip install perfreport
```

For matrix testing you also need [uv](https://docs.astral.sh/uv/) in your `PATH`:

```bash
pip install uv
# or via pipx: pipx install uv
```

## Quick start

A report is a Python module that exposes two functions: `get_args(parser)` and `get_report(options) -> Report`. Workers are Python modules invoked as subprocesses.

### 1. A minimal report

```python
# my_bench/report.py
from perfreport.models import (
    Report, ReportDefaults, WorkerModel, WorkerArgs,
    OperationModel, OpArgs, ChartModel,
)


def get_args(parser):
    group = parser.add_argument_group('My benchmark')
    group.add_argument('-H', '--hosts', default='127.0.0.1')


def get_report(options):
    worker = 'my_bench.worker'
    return Report(
        name='My benchmark',
        description='Tests how fast my_lib does X',
        default=ReportDefaults(
            worker=worker,
            options={
                'hosts': options.hosts,
                'concurrency': options.concurrency,
                'num_queries': options.num_queries,
            },
        ),
        charts=[
            ChartModel(title='Requests per second', x_key='req', y_key='rps',
                       xlabel='Requests', ylabel='RPS'),
        ],
        items=[
            Report(
                name='Scenario A',
                worker=WorkerModel(
                    args=WorkerArgs(operations=[
                        OperationModel(key='run', name='Run', args=[
                            OpArgs(key='mode', value='fast', syntax='python'),
                        ]),
                    ]),
                ),
            ),
        ],
    )
```

Reports that want the conventional `-n/--num-queries` and `-c/--concurrency` flags can plug them in via a helper:

```python
# my_bench/report.py — inside get_args:
from perfreport.args import add_common_workload_args

def get_args(parser):
    group = parser.add_argument_group('My benchmark')
    add_common_workload_args(group)  # adds -n and -c
    group.add_argument('-H', '--hosts', default='127.0.0.1')
```

### 2. A minimal Python worker

```python
# my_bench/worker.py
import sys, time, asyncio

from perfreport import WorkerArgs
from perfreport.protocol import emit_checkpoint, emit_summary, emit_environment
from perfreport.metrics import sample_cpu, sample_ram_mb
from perfreport.utils import decode_base64_pickle


async def start_test(args: WorkerArgs):
    num_queries = args.options['num_queries']
    emit_environment('MyLib', '1.0.0')
    start = time.monotonic()
    cp_time = start
    cp_interval = max(1, num_queries // 10)
    for i in range(num_queries):
        # ... do work ...
        if (i + 1) % cp_interval == 0:
            now = time.monotonic()
            elapsed = now - cp_time
            emit_checkpoint(i + 1, sample_cpu(), sample_ram_mb(), elapsed,
                            cp_interval / elapsed if elapsed else 0)
            cp_time = now
    total = time.monotonic() - start
    emit_summary(num_queries, sample_cpu(), sample_ram_mb(), total,
                 num_queries / total if total else 0)


if __name__ == '__main__':
    args: WorkerArgs = decode_base64_pickle(sys.argv[1])
    asyncio.run(start_test(args))
```

No base class to inherit from. You own the loop, own the state, emit protocol lines when ready.

### 3. A worker in another language

Any process that prints protocol lines works. Example in C:

```c
// my_bench_worker.c — compile with `cc -O2 my_bench_worker.c -o my_bench_worker`
#include <stdio.h>
#include <time.h>

int main(int argc, char** argv) {
    int n = 100000;
    printf("::environment:: Lang C\n");
    fflush(stdout);
    struct timespec t0, t1;
    clock_gettime(CLOCK_MONOTONIC, &t0);
    for (int i = 1; i <= n; ++i) {
        // ... do work ...
        if (i % (n / 10) == 0) {
            clock_gettime(CLOCK_MONOTONIC, &t1);
            double elapsed = (t1.tv_sec - t0.tv_sec) + (t1.tv_nsec - t0.tv_nsec) / 1e9;
            printf("::checkpoint:: %d 0 0 %.4f %.2f\n", i, elapsed, (n/10.0)/elapsed);
            fflush(stdout);
            t0 = t1;
        }
    }
    printf("::summary:: %d 0 0 1.0 %d\n", n, n);
    return 0;
}
```

Wire it up with `worker_cmd` in the report:

```python
Report(
    name='C worker',
    worker=WorkerModel(
        cmd='./my_bench_worker',  # bypass Python module resolution
        args=WorkerArgs(),
    ),
)
```

Same for Go, Rust, bash — anything that writes to stdout works.

### 4. Run

```bash
perfreport --report my_bench.report -n 100000 -c 100 --report-path ./out --html
```

This produces:
- `./out/report.md` — Markdown with tables, embedded SVG charts
- `./out/report.html` — HTML with interactive charts
- `./out/charts/*.svg` — per-chart SVG files

## CLI reference

Base flags (always available):

| Flag | Description |
|------|-------------|
| `--report MODULE` | Test module to run (required) |
| `--report-path DIR` | Output directory for generated report |
| `--html` | Render HTML output |
| `--offline-html` | HTML without the syntax highlighting it loads from esm.sh when opened |
| `-i, --interactive` | Interactive mode with per-scenario prompts |
| `--silent` | Suppress console output (file-only) |
| `--no-colors` | Plain ASCII output without ANSI colors |
| `--worker MODULE` | Override worker module path |
| `--worker-cmd CMD` | Override the full worker command, run without a shell; write `\\-h` for an argument `-h` |
| `--matrix-cache-dir DIR` | Cache for matrix venvs (default `.perfreport-venvs`) |
| `--rebuild-envs` | Ignore matrix venv cache |

perfreport exits with 1 when a worker failed (a non-zero exit code, or a matrix venv that could not be provisioned) or the run was stopped, the failed workers are listed on stderr and marked in the report.

The first SIGINT, SIGTERM or SIGQUIT stops the run: the running worker is terminated and the report of the workers that ran is written, marked as interrupted. A second one quits at once.

Reports contribute their own flags via `get_args(parser)`. A common set (`-n/--num-queries`, `-c/--concurrency`) can be added via `from perfreport.args import add_common_workload_args`; reports with different semantics (duration-based, num-rows, etc.) define whatever they need.

## Core concepts

### Report tree

A `Report` is a node in a hierarchical tree. Leaves have a `WorkerModel` (spawned as subprocess); branches have `items` (child reports) and optional `charts` aggregating data from descendants.

`ReportDefaults` cascades `options`, `custom_metrics`, `protocol`, `worker` path and `matrix` down the tree — children inherit, can override or extend.

### Text protocol

Workers write to stdout. The runner parses these commands:

| Command | Format | Purpose |
|---------|--------|---------|
| `::checkpoint::` | `req cpu ram time rps` | Periodic progress (default every `num_queries/10`) |
| `::summary::` | `req cpu ram time rps` | Final aggregate |
| `::environment::` | `key value` | Metadata (Python version, driver version) |
| `::custom_metrics::` | `key v1 v2 ...` | Driver-specific metrics table |

Any other stdout/stderr is captured and displayed in the `Stdout/stderr` section of the report, as is a protocol line the results do not take, with the reason.

The output is read as lines whatever chunks the pipes give, the last line may have no newline. A worker is done when it exited and its stdout and stderr are closed, its exit code is kept in `results.returncode`.

### Charts

`ChartModel` declares a chart; data source is auto-collected from sibling/descendant worker results:

```python
ChartModel(
    title='Summary RPS',
    x_key='req', y_key='rps',
    xlabel='Category', ylabel='RPS',
    data_source='summary',       # 'checkpoints' | 'summary' | 'custom'
    chart_type='bar',            # 'line' | 'bar'
    bar_sort='desc',             # 'desc' | 'asc' | 'none'
)
```

Bars auto-pivot by category when each category has one summary point.

## Matrix testing

Run the same benchmark across multiple Python versions or library versions by declaring a matrix:

```python
from perfreport.models import Matrix

default = ReportDefaults(
    worker='my_bench.worker',
    matrix=[
        Matrix(name='py311', python='3.11', deps=['my_lib==1.0.0']),
        Matrix(name='py313', python='3.13', deps=['my_lib==1.0.0']),
        Matrix(name='local-build', python='3.13',
               deps=['./dist/my_lib-1.1.0-py3-none-any.whl']),
    ],
)
```

Each matrix entry produces a parallel sub-tree. `uv` creates an isolated venv per entry, installs perfreport (its checkout editable when perfreport runs from one, else the same version from PyPI) and the deps of the entry, and runs the worker under that interpreter. Summary bar charts compare entries side-by-side.

The venv is cached by a hash of the perfreport version, python, deps and extras. A dep that is a local wheel, archive or directory counts by its size and modification time: a wheel built again at the same path gets a new venv. A worker given as a module is imported from the directory this perfreport imports it from, by `PYTHONPATH`: the report package need not be installed in the venvs.

Cache directory defaults to `./.perfreport-venvs/`. Override via `--matrix-cache-dir`; force rebuild with `--rebuild-envs`.

## Output modes

- **Default** (`VerboseHandler`) — streaming: header once, then each checkpoint appended as a single line. No screen redraws.
- **`-i, --interactive`** — full-report refresh with prompts before each worker.
- **`--silent`** — file output only.

## Writing a driver / extending

1. Put your report and worker modules under a package (e.g. `my_bench.report` and `my_bench.worker`).
2. Worker must implement `start_test(args: WorkerArgs)` as the entry point when invoked as a script.
3. Worker args (queries, values, expected results) are serialized via `encode_base64_pickle` by default: one argument that keeps UUID, Decimal, bytes and datetime, read by `decode_base64_pickle(sys.argv[1])`. `encode_typed_json` suits workers in other languages. Args longer than 100 000 chars come as `@<path>` of a file, which the decoders of perfreport read.
4. Extend `ResultsModel.custom_metrics` to capture driver-specific values (connection stats, latency percentiles, etc.).

## Development

```bash
git clone https://github.com/ssw-tech/perfreport
cd perfreport
pip install -e .[dev]
pytest
```

## License

MIT — see [LICENSE](LICENSE).
