simplebench.reporters.reporter package🔗

Reporter base package in the reporters package.

class simplebench.reporters.reporter.Reporter( config: ReporterConfig, )[source]🔗

Bases: ABC, _ReporterArgparseMixin, _ReporterOrchestrationMixin, _ReporterPrioritizationMixin, _ReporterTargetMixin, ReporterProtocol

Base class for Reporter classes.

A Reporter is responsible for generating reports based on benchmark results from a Session and Case. Reporters can produce reports in various formats and output them to different targets.

All Reporter subclasses must implement the methods defined in this interface. Reporters should handle their own output, whether to console, file system, HTTP endpoint, display device, via a callback or other output.

The Reporter interface ensures that all reporters provide a consistent set of functionalities, making it easier to manage and utilize different

reporting options within the SimpleBench framework.

add_boolean_flags_to_argparse( parser: ArgumentParser, choice: Choice, ) → None🔗

Adds a Choice’s command-line flags to an ArgumentParser.

This is a default implementation that adds boolean flags for each Choice’s flags. Subclasses can override this method if they need custom behavior such as adding arguments with different types or more complex logic.

Parameters:

parser (ArgumentParser) – The ArgumentParser to add the flags to.
choice (Choice) – The Choice instance for which to add the flags.

add_choice(choice: None) → None[source]🔗

Add a Choice to the reporter’s choices.

Parameters:

choice (Choice) – The Choice instance to add.

Raises:

SimpleBenchTypeError – If the provided choice is not a Choice instance.
SimpleBenchValueError – If the choice’s sections, targets, or formats are not supported by the reporter.

add_flags_to_argparse( parser: ArgumentParser, ) → None🔗

Add the reporter’s command-line flags to an ArgumentParser.

This is an interface method for adding flags of different types to an ArgumentParser.

Choices can define different types of flags, such as boolean flags or flags that accept multiple values (lists). This method allows adding flags of the specified type to the ArgumentParser.

Parameters:: parser (ArgumentParser) – The ArgumentParser to add the flags to.

add_list_of_targets_flags_to_argparse( parser: ArgumentParser, choice: Choice, ) → None🔗

Add a Choice’s command-line flags to an ArgumentParser.

This is a default implementation of adding flags that accept multiple values for each Choice’s flags to specify the output targets for the reporter.

Example

For a Choice with flags [’–json’], this method will add an argument to the parser that accepts multiple target values, like so:

--json: # default target

–json console filesystem callback # multiple targets –json filesystem # single target

Subclasses can override this method if they need custom behavior such as adding arguments with different types or more complex logic.

The added argument will use the ‘append’ action to allow multiple occurrences of the flag, each potentially specifying multiple targets, and uses the choices option to restrict the allowed target values to only those supported by the Choice.

This will be enforced during argument parsing, so invalid target values will result in an error.

Parameters:

parser (ArgumentParser) – The ArgumentParser to add the flags to.
choice (Choice) – The Choice instance for which to add the flags.

Raises:

SimpleBenchTypeError – If the parser arg is not an ArgumentParser instance.

property choices: Choices🔗

The Choices for the reporter.

The Choices instance contains one or more Choice instances, each representing a specific combination of sections, targets, and formats, command line flags, and descriptions.

This property allows access to the reporter’s choices for generating reports and customizing report output and available options.

Returns:: The Choices instance for the reporter.
Return type:: Choices

property config: ReporterConfig🔗: The configuration object for the reporter.

property default_targets: frozenset[Target]🔗: The default set of Targets for the reporter.

property description: str🔗: A brief description of the reporter.

Deliver the rendered output to the specified targets.

This helper method takes the rendered output and dispatches it to the appropriate targets based on the prioritized options.

This method handles the logic for delivering the report output to the possible targets: - FILESYSTEM: Writes the output to a file in the specified path and subdirectory. - CALLBACK: Sends the output to a provided callback function for further processing. - CONSOLE: Outputs the report directly to the console.

Parameters:

output (str | bytes | Text | Table) – The rendered report output.
filename_base (str) – The base filename to use for filesystem outputs. This is the filename without any suffixes or extensions.
args (Namespace) – The parsed command-line arguments.
choice (Choice) – The Choice instance specifying the report configuration.
case (Case) – The Case instance representing the benchmarked code.
section (Section) – The Section of the report.
path (Path | None) – The path to the directory where the CSV file(s) will be saved.
session (Session | None) – The Session instance containing benchmark results.
callback (ReporterCallback | None) – A callback function for additional processing of the report.

Raises:

SimpleBenchValueError – If an unsupported target is specified in the choice.

property file_append: bool🔗: Whether output files should be appended to.

property file_suffix: str🔗: The file suffix for reporter output files.

property file_unique: bool🔗: Whether output files should have unique names.

static find_options_by_type( options: Iterable[ReporterOptions] | None, cls: type[T], ) → T | None[source]🔗

Retrieve an instance of type cls (if present) from a collection of ReporterOptions.

This is used to extract reporter specific options from an iterable container of generic ReporterOptions such as those associated with a Choice or Case.

For example, a CSVReporter may define a CSVReporterOptions class that extends ReporterOptions and use this method to extract the CSVReporterOptions instance from the options iterable:

options = Reporter.find_options_by_type(case.options, CSVReporterOptions)

Parameters:

options (Iterable[ReporterOptions]) – An iterable of ReporterOptions instances.
cls (type[T]) – The specific subclass type of ReporterOptions to find.

Returns:

The instance of the class cls if found, otherwise None.

Return type:

T | None

get_all_stats_values( results: list[Results], section: Section, ) → list[float][source]🔗

Gathers all primary statistical values for a given section across multiple results.

It collects mean, median, minimum, maximum, 5th percentile, and 95th percentile, from each Results instance for the specified section.

This method is useful in determining appropriate scaling factors or units for reporting by analyzing the range of values across all results.

Adjusted standard deviation is not included in this collection because it can be NaN for results with insufficient data points, or orders of magnitude different from the other statistics, which can skew scaling calculations.

Parameters:

results (list[Results]) – A list of Results instances to gather statistics from.
section (Section) – The section to gather statistics for.

Returns:

A list of all gathered statistical values.

Return type:

list[float]

get_base_unit_for_section( section: Section, ) → str[source]🔗

Return the base unit for the specified section.

Parameters:: section (Section) – The section to get the base unit for.
Returns:: The base unit for the section.
Return type:: str

classmethod get_default_options() → ReporterOptions[source]🔗

Get the default options for the reporter.

Returns the default options set via set_default_options() if set, otherwise returns the built-in hardcoded default options from get_hardcoded_default_options().

Returns:: The default options.
Return type:: ReporterOptions

classmethod get_hardcoded_default_options() → Any[source]🔗

Get the built-in hardcoded default options for the reporter.

This abstract method must be implemented by all Reporter subclasses. It defines the base class default options for a reporter and must be available in all Reporter subclasses.

It returns the hardcoded default ReporterOptions sub-class instance specific to the reporter.

Sub-classes must implement the following class variables:

_OPTIONS_TYPE: ClassVar[type[MyOptions]] = MyOptions _OPTIONS_KWARGS: ClassVar[dict[str, Any]] = {…}

or the method will raise an exception.

Returns:: The built-in hardcoded default ReporterOptions instance.
Raises:: SimpleBenchNotImplementedError – If required class variables are not implemented or are of incorrect types.

get_prioritized_default_targets( choice: Choice, ) → frozenset[Target]🔗

Get the prioritized default targets from the choice or reporter defaults.

This method retrieves the default targets for the reporter by first checking the choice default_targets, and if none are found, falling back to the reporter’s default targets.

Parameters:: choice (Choice) – The Choice instance specifying the report configuration.
Returns:: The set of default targets.
Return type:: frozenset[Target]

get_prioritized_file_append( choice: Choice, ) → bool🔗

Get the prioritized file append flag from the choice or reporter.

This method retrieves the file append flag for the reporter by first checking the choice file_append, and if None is found falling back to the reporter’s file_append.

Parameters:: choice (Choice) – The Choice instance specifying the report configuration.
Returns:: The file append flag for report files.
Return type:: bool

get_prioritized_file_suffix( choice: Choice, ) → str🔗

Get the prioritized file suffix from the choice or reporter.

This method retrieves the file suffix for the reporter by first checking the choice file_suffix, and if None is found falling back to the reporter’s file_suffix.

Parameters:: choice (Choice) – The Choice instance specifying the report configuration.
Returns:: The file suffix for report files.
Return type:: str

get_prioritized_file_unique( choice: Choice, ) → bool🔗

Get the prioritized file unique flag from the choice or reporter.

This method retrieves the file unique flag for the reporter by first checking the choice file_unique, and if None is found falling back to the reporter’s file_unique.

Parameters:: choice (Choice) – The Choice instance specifying the report configuration.
Returns:: The file unique flag for report files.
Return type:: bool

get_prioritized_options( case: Case, choice: Choice, ) → ReporterOptions🔗

Get the reporter-specific options from the case, choice, or default options.

This method retrieves reporter-specific options of type options_cls by checking the case options first, then the choice options, and finally falling back to the reporter’s default options if none are found.

The actual type of ReporterOptions to retrieve is determined by the reporter’s options_type property - it will always be a specific subclass of ReporterOptions defined by the reporter.

Parameters:

case (Case) – The Case instance containing benchmark results.
choice (Choice) – The Choice instance specifying the report configuration.

Returns:

The prioritized instance of the class ReporterOptions. More specifically, it will be an instance of the reporter’s specific ReporterOptions subclass as defined by the reporter’s options_type property.

Return type:

ReporterOptions

Raises:

SimpleBenchNotImplementedError – If no ReporterOptions instance can be found.

get_prioritized_subdir( choice: Choice, ) → str🔗

Get the prioritized subdirectory from the choice or reporter defaults.

The prioritized subdirectory is determined by checking the Choice().subdir attribute of the provided choice argument: - If None, the reporter’s default subdir is returned. - If not None, the Choice’s subdir is returned as the prioritized subdirectory.

Semantically, a subdir of ‘’ indicates that report files should be saved in the root output directory, while any other string indicates a subdirectory within the output directory.

Parameters:: choice (Choice) – The Choice instance specifying the report configuration.
Returns:: The subdirectory for saving report files.
Return type:: str

property name: str🔗: The unique identifying name of the reporter.

property options_type: type[ReporterOptions]🔗: The specific ReporterOptions subclass associated with this reporter.

abstract render( *, case: Case, section: Section, options: ReporterOptions, ) → str | bytes | Text | Table[source]🔗

Render the report for a specific case and section.

This abstract method must be implemented by all Reporter subclasses. It is responsible for generating the actual report content for a given case and section, based on the provided options.

The output can be a string, bytes, or a Rich object (Text or Table).

Parameters:

case (Case) – The Case instance containing the benchmark results.
section (Section) – The specific Section of the results to render.
options (ReporterOptions) – The reporter-specific ReporterOptions for rendering.

Returns:

The rendered report content.

Return type:

str | bytes | Text | Table

Raises:

NotImplementedError – If the method is not implemented in a subclass.

render_by_case( *, renderer: ReportRenderer | None = None, args: Namespace, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None🔗

Render the report for an entire case at once across all applicable sections.

This method is called by the subclass’s run_report() method to run one report per case that is then processed according to the specified targets.

It calls the subclass’s render() method to actually generate the report output.

Usage of this method is appropriate when the report output encompasses all sections in a single output, such as a summary table or comprehensive report.

Usage:

from typing import TYPE_CHECKING

from simplebench.reporters.reporter.reporter import Reporter, ReporterOptions

if TYPE_CHECKING:
    from simplebench.case import Case
    from simplebench.reporters.choice.choice import Choice
    from simplebench.session import Session

class MyReporter(Reporter):
    def __init__(self, ...):
        ...

    def run_report(self, case: Case, choice: Choice, session: Session | None = None) -> None:
        self.render_by_case(renderer=self.render,
                            args=self._args,
                            case=case,
                            choice=choice,
                            path=self._path,
                            session=session,
                            callback=self._callback)

    def render(self, *, case: Case, section: Section, options: ReporterOptions) -> str:
        '''Render the report output for the entire case across all sections.'''
        ...

Parameters:

renderer (ReportRenderer | None) – The rendering function to use. If not provided, defaults to self.render.
args (Namespace) – The parsed command-line arguments.
case (Case) – The Case instance representing the benchmarked code.
choice (Choice) – The Choice instance specifying the report configuration.
path (Path | None) – The path to the directory where the CSV file(s) will be saved.
session (Session | None) – The Session instance containing benchmark results.
callback (ReporterCallback | None) – A callback function for additional processing of the report. The function should accept two arguments: the Case instance and the CSV data as a string. Leave as None if no callback is needed.

Raises:

SimpleBenchTypeError – If the provided arguments are not of the expected types or if required arguments are missing. Also raised if the callback is not callable when provided for a CALLBACK target or if the path is not a Path instance when a FILESYSTEM target is specified.
SimpleBenchValueError – If an unsupported section or target is specified in the choice.

render_by_section( *, renderer: ReportRenderer | None = None, args: Namespace, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None🔗

Render a report for each section and dispatch to targets.

This method is called by the subclass’s run_report() method to run one report per section that is then processed according to the specified targets.

It calls the subclass’s render() method to actually generate the report output.

Usage of this method is appropriate when the report output divides each case by section, such as separate files or outputs for each section of the report.

Usage:

from simplebench.reporters.reporter.reporter import Reporter, ReporterOptions

if TYPE_CHECKING:
    from simplebench.case import Case
    from simplebench.reporters.choice.choice import Choice
    from simplebench.session import Session


class MyReporter(Reporter):
    def __init__(self, ...):
        ...

    def run_report(self, case: Case, choice: Choice, session: Session | None = None) -> None:
        self.render_by_section(
                            renderer=self.render,
                            args=self._args,
                            case=case,
                            choice=choice,
                            path=self._path,
                            session=session,
                            callback=self._callback)

    def render(self, *, case: Case, section: Section, options: ReporterOptions) -> str:
        '''Render the report output for the entire case across all sections.'''
        ...

Parameters:

renderer (ReportRenderer | None) – The rendering function to use. If not provided, defaults to self.render.
args (Namespace) – The parsed command-line arguments.
case (Case) – The Case instance representing the benchmarked code.
choice (Choice) – The Choice instance specifying the report configuration.
path (Path | None) – The path to the directory where the CSV file(s) will be saved.
session (Session | None) – The Session instance containing benchmark results.
callback (ReporterCallback | None) – A callback function for additional processing of the report. The function should accept two arguments: the Case instance and the CSV data as a string. Leave as None if no callback is needed.

Raises:

SimpleBenchTypeError – If the provided arguments are not of the expected types or if required arguments are missing. Also raised if the callback is not callable when provided for a CALLBACK target or if the path is not a Path instance when a FILESYSTEM target is specified.
SimpleBenchValueError – If an unsupported section or target is specified in the choice.

report( *, args: Namespace, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None[source]🔗

Generate a report based on the benchmark results.

This method performs validation and then calls the subclass’s run_report() method.

Parameters:

args (Namespace) – The parsed command-line arguments.
case (Case) – The Case instance containing benchmark results.
choice (Choice) – The Choice instance specifying the report configuration.
path (Path | None, optional) – The path to the directory where the report can be saved if needed. Leave as None if not saving to the filesystem. Defaults to None.
session (Session | None, optional) – The Session instance containing benchmark results. Defaults to None.
callback (ReporterCallback | None, optional) – A callback function for additional processing of the report. Defaults to None.

rich_text_to_plain_text( rich_text: Text | Table, ) → str🔗

Convert Rich Text or Table to plain text by stripping formatting.

Applies a virtual console width to ensure proper line wrapping. The console width simulates how the text would appear in a terminal of the specified width.

As rich text is normally mainly used for console output, this method provides a way to convert it to plain text while preserving the intended layout as much as possible for non-console output targets.

Parameters:: rich_text (Text | Table) – The Rich Text or Table instance to convert.
Returns:: The plain text representation of the Rich Text.
Return type:: str

run_report( *, args: Namespace, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None[source]🔗

Orchestration hook for report generation.

This method is the primary customization point for controlling how a report is generated. It is called by the public report() method after all inputs have been validated.

The default implementation calls render_by_section(), which is suitable for most reporters. Subclasses can override this method to provide alternative orchestration, such as calling render_by_case() for reports that are generated once per case.

Note

This is also the correct place to implement custom logic for non-standard targets like CUSTOM.

Parameters:

args (Namespace) – The parsed command-line arguments.
case (Case) – The Case instance representing the benchmarked code.
choice (Choice) – The Choice instance specifying the report configuration.
path (Path | None, optional) – The path to the directory where report files will be saved. Defaults to None.
session (Session | None, optional) – The Session instance containing benchmark results. Defaults to None.
callback (ReporterCallback | None, optional) – A callback function for additional processing. Defaults to None.

select_targets_from_args( *, args: Namespace, choice: Choice, default_targets: Iterable[Target], ) → set[Target]🔗

Select the output targets based on command-line arguments and choice configuration.

It checks the command-line arguments for any flags corresponding to the choice and collects the specified targets. It then cross-validates the specified targets against the supported targets for the choice. If no targets are specified in the command-line arguments, the default targets are used instead.

An exception is raised if a target is specified in the arguments, or in the default targets, that are not supported by the choice.

Parameters:

args (Namespace) – The parsed command-line arguments.
choice (Choice) – The Choice instance specifying the report configuration.
default_targets (Iterable[Target]) – The default targets to use if no targets are specified in the command-line arguments.

Returns:

A set of Target enums representing the selected output targets.

Return type:

set[Target]

Raises:

SimpleBenchTypeError – If args is not an argparse.Namespace instance.
SimpleBenchValueError – If an unsupported target is specified in the arguments.

classmethod set_default_options( options: ReporterOptions | None = None, ) → None[source]🔗

Set the default options for the reporter.

Parameters:: options (ReporterOptions or None, optional) – The options to set as the default, defaults to None

property subdir: str🔗: The subdirectory where report files will be saved.

supported_formats() → frozenset[Format][source]🔗

The set of supported Format for the reporter.

This is the set of Format that the reporter can output in.

Defined Choice can only include Format that are declared in this set.

supported_sections() → frozenset[Section][source]🔗

The set of supported Section for the reporter.

This is the set of Section that the reporter can include in its reports.

Defined Choice can only include Section that are declared in this set.

supported_targets() → frozenset[Target][source]🔗

The set of supported Target for the reporter.

This is the set of Target that the reporter can output to.

Defined Choice can only include Target that are declared in this set.

target_callback( callback: ReporterCallback | None, case: Case, section: Section, output_format: Format, output: str | bytes | Text | Table, ) → None🔗

Helper method to send report data to a callback function.

Parameters:

callback (ReporterCallback | None) – The callback function to send the output to.
case (Case) – The Case instance representing the benchmarked code.
section (Section) – The Section of the report.
output_format (Format) – The Format of the report.
output (str | bytes | Text | Table) – The report data to send to the callback.

target_console( session: Session | None, output: str | bytes | Text | Table, ) → None🔗

Helper method to output report data to the console.

It uses the Rich Console instance from the Session if provided, otherwise it creates a new Console instance.

It can accept output as a string, Rich Text, or Rich Table.

Parameters:

session (Session | None) – The Session instance containing the console.
output (str | bytes | Text | Table) – The report data to print to the console.

target_filesystem( *, path: Path | None, subdir: str, filename: str, output: str | bytes | Text | Table, unique: bool, append: bool, ) → None🔗

Helper method to output report data to the filesystem.

path, subdir, and filename are combined to form the full path to the output file.

If unique is True, the filename will be made unique by prepending a counter starting from 001 to the filename and counting up until a unique filename is found. E.g. 001_filename.txt, 002_filename.txt, etc.

If append is True, the output will be appended to the file if it already exists. Otherwise, an exception will be raised if the file already exists. Note that append mode is not compatible with unique mode.

The type signature for path is Path | None because the overall report() method accepts path as Optional[Path] because it is not always required. However, this method should only be called when a valid Path is provided and will raise an exception if it is not a Path instance.

Parameters:

path (Path | None) – The path to the directory where output should be saved.
subdir (str) – The subdirectory within the path to save the file to.
filename (str) – The filename to save the output as.
output (str | bytes | Text | Table) – The report data to write to the file.
unique (bool) – If True, ensure the filename is unique by prepending a counter as needed.
append (bool) – If True, append to the file if it already exists. Otherwise, raise an error.

Raises:

SimpleBenchTypeError – If path is not a Path instance, or if subdir or filename are not strings.
SimpleBenchValueError – If both append and unique are True. Or if the output file already exists and neither append nor unique options were specified.

class simplebench.reporters.reporter.ReporterConfig( *, name: str, description: str, sections: frozenset[Section], targets: frozenset[Target], default_targets: frozenset[Target], formats: frozenset[Format], choices: ChoicesConf, file_suffix: str, file_unique: bool, file_append: bool, subdir: str, )[source]🔗

Bases: object

Immutable, attribute-based base configuration for a Reporter.

This frozen dataclass serves as the foundation for all specific reporter configuration classes (e.g., RichTableConfig). It defines the common data structure and centralizes the validation of all parameters required by reporters.

The sections, targets, and formats parameters act as master lists, constraining the values that can be used within the choices and default_targets parameters.

As a frozen dataclass, instances of this class are immutable; their state cannot be changed after creation. Validation and normalization of inputs are performed automatically in the __post_init__ method.

Attributes:: name (str): The unique name for the reporter (e.g., ‘rich-table’). description (str): A short description of what the reporter does. sections (frozenset[Section]): The master set of sections this reporter can handle. targets (frozenset[Target]): The master set of targets this reporter can output to. default_targets (frozenset[Target]): The default subset of targets to use. formats (frozenset[Format]): The master set of formats this reporter can produce. choices (ChoicesConf): Defines the reporter’s command-line interface flags. file_suffix (str): The file extension for filesystem targets, without the leading dot. file_unique (bool): If True, generate a unique filename for each output. file_append (bool): If True, append to the output file if it already exists. subdir (str): The subdirectory for saved files; '' means the root results directory.

choices: ChoicesConf🔗: A ChoicesConf object defining the reporter’s command-line interface flags. These flags allow end-users to precisely control report generation by specifying which sections to include, what output format to use, and where to send the report (targets). They can also control other reporter-specific options.

default_targets: frozenset[Target]🔗: The default subset of targets to use if not specified. Must be a subset of the main targets set.

description: str🔗: A short description of what the reporter does. Cannot be empty or blank.

file_append: bool🔗: If True, append to the output file if it already exists. Mutually exclusive with file_unique; exactly one must be True.

file_suffix: str🔗

The file extension to use for filesystem targets (e.g., ‘txt’), without the leading dot.

The suffix must consist only of alphanumeric characters and be 10 characters or less in length.

file_unique: bool🔗: If True, generate a unique filename for each output. Mutually exclusive with file_append; exactly one must be True.

formats: frozenset[Format]🔗: The master set of Format enums this reporter can produce. This constrains the formats that can be used by any ChoiceConf in the choices list.

name: str🔗: The unique name for the reporter (e.g., ‘rich-table’). Cannot be empty or blank.

sections: frozenset[Section]🔗: The master set of Section enums this reporter can handle. This constrains the sections that can be used by any ChoiceConf in the choices list.

subdir: str🔗

The subdirectory within the results directory to save files to. An empty string ('') specifies the root of the results directory (i.e., no subdirectory).

A subdirectory path should only contain valid directory name elements separated by forward slashes ('/'). Each element must consist only of alphanumeric, underscore, or dash characters, cannot be longer than 64 characters and cannot start or end with a dash or underscore.

Empty elements (i.e., consecutive slashes) are not allowed. Paths cannot start or end with a slash.

The total length of the subdirectory path must not exceed 255 characters.

targets: frozenset[Target]🔗: The master set of Target enums this reporter can output to. This constrains the targets that can be used by default_targets and any ChoiceConf in the choices list.

class simplebench.reporters.reporter.ReporterOptions[source]🔗

Bases: object

Marker base class for reporter related options.

This class serves as a base for all reporter-specific options classes used within the simplebench framework. It provides a common base type for all reporter options.

These options classes can be used to encapsulate configuration settings specific to different reporter implementations and are typically used in Choice() and Case() objects to customize reporter behavior.

simplebench.reporters.reporter package🔗

Subpackages🔗

Submodules🔗

Table of Contents

This Page