Metadata-Version: 2.4
Name: llmd-benchmark-report
Version: 0.2.1
Summary: Benchmark Report: a standard data format for LLM inference benchmark runs (schemas, validators, and converters)
License-Expression: Apache-2.0
Project-URL: Homepage, https://llm-d.ai
Project-URL: Repository, https://github.com/llm-d/llm-d-benchmark
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: pydantic>=2.0
Requires-Dist: PyYAML
Requires-Dist: numpy

# Benchmarking Report

A benchmarking report is a standard data format describing the cluster configuration, workload, and results of a benchmark run. The report acts as a common API for different benchmarking experiments. Each supported harness in llm-d-benchmark creates a benchmark report upon completion of a run, in addition to saving results in its native format.

There are two lines of the benchmark report, `0.1` and `0.2`. All reports are generated by the `llm-d-benchmark` harness pod, but new applications consuming benchmark data should use the `0.2` line, whose current revision is `0.2.1`. Reports that declare `0.2` were produced before that revision and still load: every valid `0.2` report is also a valid `0.2.1` report.

## Installation

The library is distributed on PyPI as [`llmd-benchmark-report`](https://pypi.org/project/llmd-benchmark-report/):

```bash
pip install llmd-benchmark-report
```

The canonical import name is `llmd_benchmark_report`. Two aliases exist for backwards compatibility and will be removed in a future release: the bare `benchmark_report` name (shipped in this distribution, used historically by the harness pod scripts) and `llmdbenchmark.analysis.benchmark_report` (a shim in the `llmdbenchmark` package). Both alias the canonical modules with identity preserved.

## Versioning and compatibility

`llmd-benchmark-report` package versions track the Benchmark Report schema version: package `0.2.x` implements BR v0.2.x (and still parses earlier report versions). Within a minor line, releases are strictly additive: new optional fields and new component kinds may be added, but existing fields are never removed, renamed, retyped, or given stricter validation that rejects previously valid reports. A report valid against `0.2.x` validates against every later `0.2.y`. Breaking changes require a new minor line. Each minor line has a single implementation: `schema_v0_2.py` and `native_to_br0_2.py` hold the latest `0.2.x` revision, and a new revision changes them in place rather than adding modules. Packaging-only fixes with no schema change ship as post-releases (e.g. `0.2.1.post1`). Consumers should depend on `llmd-benchmark-report>=0.2.1,<0.3`.

## Releasing

Releases are published to PyPI by the `br-release` GitHub workflow, triggered by pushing a `br-v*` tag (e.g. `br-v0.2.1`) whose version matches `version` in this directory's `pyproject.toml`; the workflow fails on a mismatch. Publishing uses PyPI [trusted publishing](https://docs.pypi.org/trusted-publishers/) (OIDC), so no credentials are stored in the repository.

## v0.2.x Format Description

A benchmark report describes the inference service configuration, workload, and performance results. Individual traces from single inference executions are not captured, but time-series or statistical results may be captured.

See [`br_v0_2_example.yaml`](llmd_benchmark_report/br_v0_2_example.yaml) for a dummy example report (values may be non-sensical). A [JSON Schema](https://json-schema.org/draft/2020-12) for the current `0.2.x` revision is at [`br_v0_2_json_schema.json`](llmd_benchmark_report/br_v0_2_json_schema.json). 

The top-level fields for the benchmark report are `version`, `run`, `scenario`, and `results`.

### `version` Field

The `version` field (`0.2.1` for reports produced by this package, `0.2` for reports produced before that revision) indicates the schema revision for the benchmark report. Different versions may add or drop certain details, so there is no guarantee of "lossless" conversion between benchmark report versions.

### `run` Field

The `run` field contains details about the benchmark run. Only the unique ID `run.uid` is required, with other fields including start/end times and duration, user name, and additional IDs for the experiment (to group benchmark reports), cluster, and pod.

Two fields are populated from the experiment ID (`LLMDBENCH_RUN_EXPERIMENT_ID`, or the results directory name for an archived run):

- `run.eid` groups every report of one experiment, as a UUID5 of the experiment ID. Stages of the same run share it; treatments of a sweep do not.
- `run.description` is the human-readable run label. Consumers such as [llm-d-prism](https://github.com/llm-d/llm-d-prism) read it as the run label, so it defaults to `<model> [<experiment id>]` (for example `Qwen/Qwen3-32B [inference-perf-conc32-1786024743-hipkpq]`): the ID keeps the treatments of a sweep from collapsing into one entry, and the model keeps the label readable. When no experiment ID is available at all, both fields are omitted.

The schema documents `run.description` as submitter-provided, so the generated label is only a default. Set your own with `--run-description`, `LLMDBENCH_DESCRIPTION_TEXT`, or `description.text` under a scenario's `common:` block, and it is used verbatim. A submitter-supplied description and keywords are recorded even when no experiment ID resolves, since neither needs one to be meaningful.

`run.keywords` is populated only from `--run-keywords`, `LLMDBENCH_DESCRIPTION_KEYWORDS`, or `description.keywords` (comma-separated). It is never auto-populated -- see [SUBMISSION_POLICY.md](../llmdbenchmark/results_store/SUBMISSION_POLICY.md) -- and the key is omitted entirely when the submitter sets nothing.

### `scenario` Field

`scenario` describes the inference stack configuration (under `scenario.stack`) and workload details (under `scenario.load`). These are the inputs to a benchmark experiment, and ideally will be sufficiently populated to ensure the benchmark is reproducible and leaves no question as to what exactly is being measured.

`scenario.stack` consists of a list of components such as inference engine instances (vLLM, SGLang...), routers, and any other part of the system used to support the inference endpoint. No specific component is required, so this may be customized to the needs of a particular experiment.

Each component has a `metadata`, `standardized`, and `native` field. The `native` subsection can store any arguments, environment variables, and configuration details in the native format of a particular component. The `standardized` subsection stores details in a single standardized schema for that component, regardless of the specific vendor or version used. For example, an inference engine component could be vLLM, SGLang, etc. but they will always certain configuration options in common albeit defined in different ways. The `standardized` subsection for an inference engine contains common details like LLM model name, accelerator hardware model and count, and parallelisms used.

The workload details under `scenario.load` follow the same methodology as components, with `metadata`, `standardized`, and `native` subsections.

### `results` Field

The `results` field contains outputs from a benchmarking experiment. This may include any one of request-level performance metrics, session-level performance metrics, observability data, stack component health, or profiling.

#### `results.request_performance`

Aggregate statistics for individual inference requests: latencies (TTFT, TPOT, ITL, E2E), throughput (input/output/total token rates, request rate), and request counts.

#### `results.session_performance`

Session-level statistics for multi-turn workloads. Populated from `*_session_lifecycle_metrics.json` files produced by `inference-perf` when multi-turn sessions are enabled. The `session_performance.sessions` object contains:

| Field | Description |
|---|---|
| `total` | Total number of sessions run |
| `succeeded` / `failed` | Session outcome counts |
| `total_events` / `total_events_completed` / `total_events_cancelled` | Event (per-turn request) counts |
| `session_rate` | Rate of session completions per second (`Statistics`) |
| `session_duration` | Distribution of session wall-clock durations in seconds (`Statistics`) |
| `events_per_session` | Distribution of event counts per session (`Statistics`) |
| `events_cancelled_per_session` | Distribution of cancelled event counts per session (`Statistics`) |
| `input_tokens_per_session` | Distribution of total input tokens per session (`Statistics`) |
| `output_tokens_per_session` | Distribution of total output tokens per session (`Statistics`) |

A `session_performance` report is generated alongside the standard `request_performance` report for each stage that has a corresponding session lifecycle file. The `scenario.load.standardized.stage` field identifies which stage the report covers, and `scenario.load.standardized.multi_turn.enabled` is set to `true`.

### Additions in `0.2.1`

Revision `0.2.1` adds optional multi-modal payload statistics for image, video, and audio workloads, plus optional treatment-grouping metadata on `scenario.load.metadata` (`treatment`, `treatment_group`, `concurrent_with`). `concurrent_with` names the treatments that ran at the same time against the same endpoint: when it is non-empty the metrics reflect contention from those workloads rather than this one in isolation. Every field introduced is optional, so any `0.2` report validates unchanged under `0.2.1` (enforced by `tests/test_benchmark_report_v0_2_legacy.py`).

The multi-modal additions, all derived from what the client can determine from the payloads it sent, are:

- **`results.request_performance.aggregate.requests.request_size`** (`Statistics`): total encoded request size in bytes, capturing the large payloads typical of multi-modal requests.
- **`results.request_performance.aggregate.requests.multimodal`**: a per-modality block (`image`, `video`, `audio`), each a distribution set over the media instances sent. `image`/`video` carry `count`, `filesize`, `pixels`, and `aspect_ratio`; `video` adds `frames`; `audio` carries `count`, `filesize`, and `duration`.
- **`results.request_performance.aggregate.throughput.{image,video,audio}_rate`** (`Statistics`): per-modality delivery rates (`images/s`, `videos/s`, `audios/s`).

New unit categories back these fields: `pixels` (quantity), `ratio` (for aspect ratio, distinct from a 0..1 portion), `bytes` (memory), and a media-throughput category (`images/s`, `videos/s`, `audios/s`) kept separate from the request-rate category so the existing `request_rate` guardrail is unaffected.

Server-side multi-modal metrics (vision token counts, encoding time, multimodal cache hit rates) are out of scope for this revision.

## v0.1 Format Description

A benchmark report describes the inference service configuration, workload, and aggregate results. Individual traces from single inference executions are not captured, rather statistics from multiple traces of identical scenarios are combined to create a report.

See [`br_v0_1_example.yaml`](llmd_benchmark_report/br_v0_1_example.yaml) for a dummy example report (values may be non-sensical). The [JSON Schema](https://json-schema.org/draft/2020-12) for the benchmark report version `0.1` is at [`br_v0_1_json_schema.json`](llmd_benchmark_report/br_v0_1_json_schema.json). The report has three top-level fields, `version`, `scenario`, and `metrics`.

While each of these fields is required, some subfields may be optional or not apply to the specific benchmark being performed. For example, some metrics may not be captured or supported by a certain benchmarking toolset. In cases where one desires to capture information that is not part of the standard benchmark report schema, a `metadata` field may be placed almost anywhere under `scenario` or `metrics` to add arbitrary data.

### `version` Field

The `version` field (`0.1`) indicates the schema revision for the benchmark report. Different versions may add or drop certain details, so there is no guarantee of "lossless" conversion between benchmark report versions.

### `scenario` Field

The `scenario` describes precisely what was measured. This includes details about the inference platform (full stack, including versions and the important runtime arguments), cluster configuration (like GPUs and parallelism utilized), and workload. The content in this field should be detailed enough that it is sufficient to launch a repeat benchmarking experiment that will yield similar results (within some reasonable bound of variability).

> [!NOTE]
> With future revisions the `scenario` field could be used as an input format for executing benchmark runs. For this to be practical we would first need to standardize the definition of a workload, specifically in `scenario.load.args` which is currently the exact arguments (non-standard) sent to a particular harness. In addition, the input format would need a way to describe swept parameters. A benchmark report currently describes a single point in any sweep.

### `metrics` Field

The `metrics` field contains all of the results for the report. This does not include individual trace details, rather statistics for all runs that were captured in benchmarking for a particular scenario. This includes request-level performance metrics (like latencies and throughput), details about the inference service (like request queue lengths and KV cache size), and hardware metrics (such as GPU compute and memory utilization). Some of the underlying fields, such as the hardware metrics, more traditionally fall in the category of "observability" rather than "benchmarking".


## Implementation and Usage

The schema for a benchmarking report is defined through Python classes using [Pydantic](https://docs.pydantic.dev/latest/) in `schema_{version}.py`. Instantiating an instance of a benchmark report class includes various checks, such as ensuring compliance with the schema, proper use of units, and defining all required entities.

### Requirements

```
numpy>=2.3.1
pydantic>=2.11.7
PyYAML>=6.0.2
```

### Creating a `BenchmarkReport`

A `BenchmarkReport` class may be instantiated from a JSON/YAML file with `import_benchmark_report()`, or from a JSON/YAML string with `yaml_str_to_benchmark_report()`. Both of these functions will detect the appropriate version of the benchmark report.

To generate a JSON or YAML string from a `BenchmarkReport` instance, use the `get_json_str()` and `get_yaml_str()` methods. To save as a JSON/YAML file, use the `export_json()` or `export_yaml()` methods.

```python
import llmd_benchmark_report

# Instantiate a BenchmarkReport instance from a YAML file
br = llmd_benchmark_report.import_benchmark_report("llmd_benchmark_report/br_v0_2_example.yaml")

# Create a YAML string from the BenchmarkReport
br_yaml_str = br.get_yaml_str()

# Print the string
print(br_yaml_str)

# Save the BenchmarkReport as a YAML file
br.export_yaml("br_test.yaml")
```

An instance of `BenchmarkReport` may be created directly, for example:

```python
import llmd_benchmark_report

br = llmd_benchmark_report.BenchmarkReportV02(
    **{
        "run": {
            "uid": "38b1f169ca178b756f7483523b17de61",
        },
        "scenario": {
            "stack": [
                {
                    "metadata": {
                        "schema_version": "0.0.1",
                        "label": "vllm-svc-0",
                        "cfg_id": "cc73fc6b51a1d3b8128f312d70476d7c",
                    },
                    "standardized": {
                        "kind": "inference_engine",
                        "tool": "llm-d",
                        "tool_version": "ghcr.io/llm-d/llm-d-cuda:0.3.1",
                        "role": "decode",
                        "replicas": 2,
                        "model": {
                            "name": "Qwen/Qwen3-0.6B",
                        },
                        "accelerator": {
                            "model": "NVIDIA-H100-80GB-HBM3",
                            "count": 8,
                            "parallelism": {
                                "tp": 8,
                            },
                        },
                    },
                    "native": {},
                },
            ],
            "load": {
                "metadata": {
                    "schema_version": "0.0.1",
                    "cfg_id": "a4e18f265cc33786a42b8a3f7ac2edcb",
                },
                "standardized": {
                    "tool": "inference-perf",
                    "tool_version": "0.3.0",
                    "input_seq_len": {
                        "distribution": "fixed",
                        "value": 2148,
                    },
                    "source": "sampled",
                },
                "native": {},
            },
        },
        "results": {
            "request_performance": {
                "aggregate": {
                    "latency": {
                        "request_latency": {
                            "units": "s",
                            "mean": 0.0368578234128654,
                        },
                    },
                },
            },
        },
    }
)

print(br.get_yaml_str())
```

The native formats returned by different harnesses may be converted to a benchmark report using functions in [native_to_br0_2.py](llmd_benchmark_report/native_to_br0_2.py), or using the CLI defined in `cli.py`.

To run the converter CLI, execute the module using `python -m` from the root of this repository:

```bash
benchmark-report \
  --workload-generator <generator_name> \
  --br-version <version> \
  <path_to_raw_results_file> \
  [path_to_output_report_yaml]
```

#### Parameters Reference
* `-w, --workload-generator`: Specifies the harness generator. Must be one of: `'guidellm'`, `'inferencemax'`, `'inference-perf'`, `'vllm-benchmark'`, `'nop'`.
* `-b, --br-version`: Target benchmark report version (defaults to `0.1`; `0.2` and `0.2.1` both produce the current `0.2.x` revision, `0.2.1`).
* `-f, --force`: Overwrites the output file if it already exists.
* `-i, --index`: Convert only the benchmark at this index, for a results file holding several (see [Multi-benchmark results files](#multi-benchmark-results-files-guidellm) below). Omit it to convert all of them.
* `results_file` *(Positional)*: Path to the raw native results file to convert. (e.g. For `inference-perf`, this must contain `"stage_"` in its filename, e.g., `stage_0_lifecycle_metrics.json`).
* `output_file` *(Positional, Optional)*: Destination for the converted report. If omitted, the YAML output is printed directly to `stdout`.

Keep the two positionals together rather than separating them with a flag. On Python 3.11, argparse cannot match a positional that follows an optional, so `output_file` is rejected as an unrecognized argument; Python 3.12 and later parse it correctly. The harness pod image is Python 3.13, but `requires-python` is `>=3.11`, so a local checkout may hit this:

```bash
# Portable: positionals adjacent, either before or after the flags.
... cli -w guidellm -b 0.2 results.json benchmark_report.yaml
... cli results.json benchmark_report.yaml -w guidellm -b 0.2

# Python 3.11: "error: unrecognized arguments: benchmark_report.yaml"
... cli results.json -w guidellm -b 0.2 benchmark_report.yaml
```

The output is always YAML, whatever the output filename's extension. To write JSON, use `export_json()` from the Python API instead ([Creating a `BenchmarkReport`](#creating-a-benchmarkreport)); a report's YAML can also be converted after the fact with `yq -o=json`.

#### Example
To convert a raw `stage_0_lifecycle_metrics.json` results file from an `inference-perf` run and save the v0.2 report to `benchmark_report_v02.yaml`:

```bash
benchmark-report \
  -w inference-perf \
  -b 0.2 \
  ./stage_0_lifecycle_metrics.json \
  ./benchmark_report_v02.yaml
```

---

#### Multi-benchmark Results Files (GuideLLM)

GuideLLM writes a single `results.json` holding every benchmark of a run -- one per profile stage, so a `constant` profile with `rate: [2, 4]` yields two. A benchmark report describes one benchmark, so the CLI writes one report per stage, inserting the index before the extension of the name given:

```bash
# Writes benchmark_report_0.yaml and benchmark_report_1.yaml
benchmark-report \
  -w guidellm -b 0.2 results.json benchmark_report.yaml
```

With no output file, all of them are printed to `stdout`, each preceded by a `# Benchmark <n> of <total>` comment.

Pass `-i` / `--index` to convert a single stage instead. The report is then written to the given filename unchanged, with no index suffix:

```bash
# Writes benchmark_report.yaml, for the first stage only
benchmark-report \
  -w guidellm -b 0.2 -i 0 results.json benchmark_report.yaml
```

Which stage a report came from is recorded in `scenario.load.standardized.stage`, alongside the `rate_qps` or `concurrency` that stage actually ran at.

#### Session Lifecycle Files (Inference-Perf)
For `inference-perf` session lifecycle files (`*_session_lifecycle_metrics.json`), pass the `-s` / `--session` flag:

```bash
# Print to stdout
benchmark-report -w inference-perf -b 0.2 -s stage_0_session_lifecycle_metrics.json

# Save to file
benchmark-report -w inference-perf -b 0.2 -s stage_0_session_lifecycle_metrics.json benchmark_report_session.yaml
```

The `-s` flag selects the `import_inference_perf_session` converter, which populates `results.session_performance` instead of `results.request_performance`. The `inference-perf-analyze_results.sh` script detects `*_session_lifecycle_metrics.json` files automatically and passes `-s` without manual intervention.
