simplebench.reporters.json.reporter package🔗

JSON Reporter package for simplebench.

class simplebench.reporters.json.reporter.JSONConfig(
*,
name: str | None = None,
description: str | None = None,
sections: Iterable[Section] | None = None,
targets: Iterable[Target] | None = None,
default_targets: Iterable[Target] | None = None,
formats: Iterable[Format] | None = None,
choices: ChoicesConf | None = None,
file_suffix: str | None = None,
file_unique: bool | None = None,
file_append: bool | None = None,
subdir: str | None = None,
)[source]🔗

Bases: ReporterConfig

Configuration for a JSONReporter.

This class inherits from ReporterConfig and provides a type-safe, discoverable interface for overriding the default settings of a JSONReporter.

class simplebench.reporters.json.reporter.JSONOptions(*, full_data: bool = False)[source]🔗

Bases: ReporterOptions

Class for holding JSON reporter specific options in a Choice or Case.

This class provides additional configuration options specific to the JSON reporter. It is accessed via the options attribute of a Choice or Case instance.

Variables:

full_data (bool) – Whether to include full data in the JSON output.

property full_data: bool🔗

Return whether to include full data in the JSON output.

Returns:

Whether to include full data in the JSON output.

Return type:

bool

class simplebench.reporters.json.reporter.JSONReporter(
config: JSONConfig | None = None,
)[source]🔗

Bases: Reporter

Class for outputting benchmark results to JSON files.

It supports reporting statistics for various sections, either separately or together, to the filesystem, via a callback function, or to the console in JSON format.

The JSON files are tagged with metadata comments including the case title, description, and units for clarity.

Defined command-line flags:

  • --json: {filesystem, console, callback} (default=filesystem) Outputs statistical results to JSON.

  • --json-data: {filesystem, console, callback} (default=filesystem) Outputs results to JSON with full data.

Example usage:

program.py --json               # Outputs results to JSON files in the filesystem (default).
program.py --json filesystem    # Outputs results to JSON files in the filesystem.
program.py --json console       # Outputs results to the console in JSON format.
program.py --json callback      # Outputs results via a callback function in JSON format.
program.py --json filesystem console  # Outputs results to both JSON files and the console.
Variables:

  • name (str) – The unique identifying name of the reporter.

  • description (str) – A brief description of the reporter.

  • choices (Choices) – A collection of Choices instances defining the reporter instance, CLI flags, Choice name, supported Section objects, supported output Target objects, and supported output Format objects for the reporter.

  • targets (set[Target]) – The supported output targets for the reporter.

  • formats (set[Format]) – The supported output formats for the reporter.

mean_change(
first: Case,
second: Case,
section: Section,
) → float | None[source]🔗

Compare two Case instances for a given section and return the change as a float ratio.

The float ratio is calculated as (value2 - value1) / value1, where value1 and value2 are the mean values for the specified section in the first and second cases, respectively.

A value of 0.0 indicates no change, a positive value indicates an increase, and a negative value indicates a decrease. If either case does not have data for the specified section, None is returned. The ratio is limited to 3 significant digits for clarity and to prevent a false sense of precision to the result.

If first mean value is 0.0 and second mean value is also 0.0, the change is defined as 0.0. If first mean value is 0.0 and second mean value is non-zero, the change is defined as None to indicate incomparability.

Parameters:

  • first – The first Case instance to compare.

  • second – The second Case instance to compare.

  • section – The Section to compare.

Returns:

The change between the mean for two cases for the specified section, or None if the section is not present in either case or the numbers are incomparable.

render(
*,
case: Case,
section: Section,
options: ReporterOptions,
) → str[source]🔗

Convert the Case data for all sections to a JSON string.

Machine info is included in the JSON output under the ‘metadata’ key.

Parameters:

  • case – The Case instance holding the benchmarked code statistics.

  • section – The Section to render (ignored, all sections are included).

  • options – The JSONOptions instance specifying rendering options or None if not provided. (JSONOptions is a subclass of ReporterOptions.)

Returns:

The JSON string representation of the Case data.

run_report(
*,
args: Namespace,
case: Case,
choice: Choice,
path: Path | None = None,
session: Session | None = None,
callback: ReporterCallback | None = None,
) → None[source]🔗

Output the benchmark results to a file as tagged JSON if available.

This method is called by the base class’s report() method after validation. The base class handles validation of the arguments, so subclasses can assume the arguments are valid without a large amount of boilerplate code. The base class also handles lazy loading of the reporter classes, so subclasses can assume any required imports are available.

The run_report() method’s main responsibilities are to select the appropriate output method (render_by_case() in this case) based on the provided arguments and to pass the actual rendering method to be used (the render() method in this case). The rendering method must conform with the ReportRenderer protocol.

Parameters:

  • args – The parsed command-line arguments.

  • case – The Case instance representing the benchmarked code.

  • choice – The Choice instance specifying the report configuration.

  • path – The path to the directory where the JSON file(s) will be saved.

  • session – The Session instance containing benchmark results.

  • callback – A callback function for additional processing of the report. The function should accept two arguments: the Case instance and the JSON data as a string. Leave as None if no callback is needed.

Subpackages🔗

Submodules🔗