simplebench package🔗

Simple benchmarking framework.

class simplebench.CSVOptions[source]🔗

Bases: ReporterOptions

Class for holding CSV reporter specific options.

This class provides additional configuration options specific to the CSV reporter. It is accessed via the options attribute of a Choice instance or a Case instance.

It is currently only a stub for future expansion.

class simplebench.Case(
*,
action: ActionRunner,
group: str = 'default',
title: str | None = None,
description: str | None = None,
iterations: int = 20,
warmup_iterations: int = 10,
rounds: int = 1,
min_time: float = 5.0,
max_time: float = 20.0,
variation_cols: dict[str, str] | None = None,
kwargs_variations: dict[str, list[Any]] | None = None,
runner: type[SimpleRunner] | None = None,
callback: ReporterCallback | None = None,
options: Iterable[ReporterOptions] | None = None,
)[source]🔗

Bases: object

Declaration of a benchmark case.

A benchmark case defines the specific benchmark to be run, including the action to be performed, the parameters for the benchmark, and any variations of those parameters as well as the reporting group and title for the benchmark.

It also defines the number of iterations, warmup iterations, rounds, minimum and maximum time for the benchmark, the benchmark runner to use, and any callbacks to be invoked to process the results of the benchmark for reporting purposes.

The min_time, max_time, iterations, and warmup_iterations parameters control how the benchmark is executed and measured and interact with each other as follows when using the default SimpleRunner: - The benchmark will perform warmup_iterations iterations before starting the timing

and measurement phase. This is done to allow for any setup or caching effects to stabilize. This is separate from the main benchmark iterations and does not count towards the iterations count or the min_time/max_time limits.

  • The benchmark will run for at least min_time wall clock seconds, but will stop on

    completing the first iteration that ends after max_time seconds during the timing phase.

  • If the benchmark completes iterations iterations after min_time but before

    reaching max_time, it will stop.

This means that the benchmark will run for at least min_time seconds and for at least one iteration during the timing phase. If min_time is reached before iterations is completed, the benchmark will continue running until either iterations or max_time is completed (whichever happens first).

rounds specifies the number of times the action will be executed per iteration to get a better average. Each iteration will run the specified number of rounds after setup and before teardown. The timing for the iteration will be the average time taken for the rounds in that iteration.

This helps to reduce the impact of variability in execution time for a single run of the action for very fast actions. This suppresses the overhead of the loop and timer quantization in Python during the actual timing benchmark/measurement phase. Internally, the action is called rounds times in an unrolled loop for each iteration, and the average time per call is used for the iteration timing.

This removes the overhead of the loop and timer quantization in Python during the actual timing benchmark/measurement phase by aggregating multiple calls to the action within a single iteration without the overhead of looping constructs. This allows for more accurate timing of very fast actions by reducing the relative impact of loop overhead and timer resolution limitations.

The trade-off is that total number of action calls is now iterations * rounds, and the reported time per action call is an average over the rounds in each iteration. This can dramatically improve the accuracy of timing measurements for very fast actions, at the cost of increased total execution time for the benchmark due to the additional calls to the action.

The unrolled loop means that setup and teardown functions (if any) are called only once per iteration, not once per round. All rounds in the same iteration share the same setup/teardown context.

If your action is not extremely fast (~ 10 nanoseconds or faster), it is recommended to leave rounds at its default value of 1. If you do use it, you may want to run dual benchmarks with rounds=1 and rounds>1 to see how much the reported variability and other metrics change.

The Case class is designed to be immutable after creation. Once a Case instance is created, its properties cannot be directly changed. This immutability ensures that benchmark cases remain consistent throughout their lifecycle.

The results of the benchmark runs are stored in the results property, which is a list of Results objects. Each Results object corresponds to a specific combination of keyword argument variations.

Minimal Example🔗
  from simplebench import (
      Case, SimpleRunner, Results, main)


  def my_benchmark_action(bench: SimpleRunner,
                          **kwargs) -> Results:
      # Perform benchmark action here
      def benchmark_operation():
          sum(range(1000))  # Example operation to benchmark

      return bench.run(benchmark_operation)


  if __name__ == '__main__':
      cases_list: list[Case] = [
          Case(action=my_benchmark_action)
      ]
      main(cases_list)
property action: ActionRunner🔗

The function to perform the benchmark.

The function must accept a bench parameter of type SimpleRunner and arbitrary keyword arguments (’**kwargs’) and return a Results object.

Example:

def my_benchmark_action(*, bench: SimpleRunner, **kwargs) -> Results:
    def setup_function(size: int) -> None:
        # Setup code goes here
        pass

    def teardown_function(size: int) -> None:
        # Teardown code goes here
        pass

    def action_function(size: int) -> None:
        # The code to benchmark goes here
        lst = list(range(size))

    # Perform the benchmark using the provided SimpleRunner instance
    results: Results = bench.run(
        n=kwargs.get('size', 1),
        setup=setup_function, teardown=teardown_function,
        action=action_function, **kwargs)
    return results
as_dict(
full_data: bool = False,
) → dict[str, Any][source]🔗

Returns the benchmark case and results as a JSON serializable dict.

Only the results statistics are included by default. To include full results data, set full_data to True.

Parameters:

full_data (bool) – Whether to include full results data. Defaults to False.

Returns:

A JSON serializable dict representation of the benchmark case and results.

Return type:

dict[str, Any]

property callback: ReporterCallback | None🔗

A callback function for additional processing of a report.

A callback function to be called with the benchmark results in a reporter. This function should accept four arguments: the Case instance, the Section, the ReporterOption, and the output object. Leave as None if no callback is needed. (default: None)

property description: str🔗

A brief description of the benchmark case.

If not specified, defaults to the docstring of the action function or ‘(no description)’ if no docstring is available.

Cannot be blank.

property expanded_kwargs_variations: list[dict[str, Any]]🔗

All combinations of keyword arguments from the specified kwargs_variations.

A mapping of keyword argument names to their variations.

Each key is a keyword argument name, and the value is a list of possible values.

When tests are run, the benchmark will be executed for each combination of the specified keyword argument variations. For example, if kwargs_variations is

kwargs_variations argument example🔗
  ...
  kwargs_variations = {
          'size': [10, 100],
          'mode': ['fast', 'accurate']
      },
  ...

The benchmark will be run 4 times with the following combinations of keyword arguments:

Keyword (**kwargs) Argument Combinations🔗
1  {size=10, mode='fast'}
2  {size=10, mode='accurate'}
3  {size=100, mode='fast'}
4  {size=100, mode='accurate'}

The action function will be called with these keyword arguments accordingly and must accept them.

Returns:

A list of dictionaries, each representing a unique combination of keyword arguments.

Return type:

list[dict[str, Any]]

property group: str🔗

The benchmark reporting group to which the benchmark case belongs for selection and reporting purposes.

Cannot be blank. It is used to categorize and filter benchmark cases.

property iterations: int🔗

The number of iterations to run for the benchmark.

property kwargs_variations: dict[str, list[Any]]🔗

Variations of keyword arguments for the benchmark.

Each key is a keyword argument name, and the value is the column label to use for that argument. Only keywords that are also in kwargs_variations can be used here. These fields will be added to the output of reporters that support them as columns of data with the specified labels.

When tests are run, the benchmark will be executed for each combination of the specified keyword argument variations. For example, if kwargs_variations is

kwargs_variations argument example🔗
  ...
  kwargs_variations = {
          'size': [10, 100],
          'mode': ['fast', 'accurate']
      },
  ...

The benchmark will be run 4 times with the following combinations of keyword arguments:

Keyword (**kwargs) Argument Combinations🔗
1  {size=10, mode='fast'}
2  {size=10, mode='accurate'}
3  {size=100, mode='fast'}
4  {size=100, mode='accurate'}

The action function will be called with these keyword arguments accordingly and must accept them.

property max_time: float🔗

The maximum time for the benchmark in seconds.

property min_time: float🔗

The minimum time for the benchmark in seconds.

property options: list[ReporterOptions]🔗

A list of additional options for the benchmark case.

property results: list[Results]🔗

The benchmark list of Results for the case.

This is a read-only attribute. To add results, use the run method.

Returns:

A list of Results objects for each variation run of the benchmark case.

Return type:

list[Results]

property rounds: int🔗

The number of rounds to run for the benchmark for each iteration.

Rounds are multiple runs of the entire benchmark to get a better average for an iteration. Each iteration will run the specified number of rounds after setup and before teardown. (default: 1)

run(
session: Session | None = None,
) → None[source]🔗

Run the benchmark tests.

This method will execute the benchmark for each combination of keyword arguments and collect the results. After running the benchmarks, the results will be stored in the self.results attribute.

If passed, the session’s tasks will be used to display progress, control verbosity, and pass CLI arguments to the benchmark runner.

Parameters:

session (Optional[Session]) – The session to use for the benchmark case.

property runner: type[SimpleRunner] | None🔗

A custom runner class for the benchmark.

If None, the default SimpleRunner is used. (default: None)

A custom runner class must be a subclass of SimpleRunner and must have a method named run that accepts the same parameters as SimpleRunner.run and returns a Results object. The action function will be called with a bench parameter that is an instance of the custom runner.

It may also accept additional parameters to the run method as needed. If additional parameters are required, they must be specified in the action function signature.

section_mean(
section: Section,
) → float[source]🔗

Calculate the mean value for a specific section across all results.

This method computes the mean value for the specified section (either OPS or TIMING) across all benchmark results associated with this case.

This is a very ‘hand-wavy’ mean calculation that simply averages the means of each result. It does not take into account the number of iterations or other statistical factors. It is intended to provide a rough estimate of the overall performance for the specified section for use in comparisons between successive benchmark runs in tests looking for large performance regressions. As such, it should not be used for any rigorous statistical analysis.

Parameters:

section (Section) – The section for which to calculate the mean.

Returns:

The mean value for the specified section.

Return type:

float

property title: str🔗

The name of the benchmark case.

If not specified, defaults to the name of the action function. Cannot be blank.

static validate_action_signature(
action: ActionRunner,
) → ActionRunner[source]🔗

Validate that action has correct signature.

An action function must accept the following two parameters:

  • bench: SimpleRunner

  • **kwargs: Arbitrary keyword arguments

This is equivalent to the ActionRunner protocol.

Parameters:

action (ActionRunner) – The action function to validate.

Returns:

The validated action function.

Return type:

ActionRunner

Raises:

SimpleBenchTypeError – If the action is not callable or has an invalid signature.

static validate_kwargs_variations(
value: dict[str, list[Any]] | None,
) → dict[str, list[Any]][source]🔗

Validate the kwargs_variations dictionary.

Validates that the kwargs_variations is a dictionary where each key is a string that is a valid Python identifier, and each value is a non-empty list.

A shallow copy of the validated dictionary and the lists is performed before returning to prevent external modification.

Parameters:

value (dict[str, list[Any]] | None) – The kwargs_variations dictionary to validate. Defaults to {} if None.

Returns:

A shallow copy of the validated kwargs_variations dictionary or {} if not provided. The keys are strings that are valid Python identifiers, and the values are non-empty lists. The lists may contain any type of values.

Return type:

dict[str, list[Any]]

Raises:

  • SimpleBenchTypeError – If the kwargs_variations is not a dictionary or if any key is not a string that is a valid Python identifier.

  • SimpleBenchValueError – If any value is not a list or is an empty list.

static validate_options(
value: Iterable[ReporterOptions] | None,
) → list[ReporterOptions][source]🔗

Validate the options list.

Parameters:

value (Iterable[ReporterOption] | None) – The options iterable to validate or None.

Returns:

A shallow copy of the validated options as a list or an empty list if not provided.

Return type:

list[ReporterOptions]

Raises:

SimpleBenchTypeError – If options is not a list or if any entry is not a ReporterOption.

static validate_runner(
value: type[SimpleRunner] | None,
) → type[SimpleRunner] | None[source]🔗

Validate the runner class.

Parameters:

value (Optional[type[SimpleRunner]]) – The runner class to validate.

Returns:

The validated runner class or None.

Return type:

Optional[type[SimpleRunner]]

Raises:

SimpleBenchTypeError – If the runner is not a subclass of SimpleRunner or None.

static validate_time_range(
min_time: float,
max_time: float,
) → None[source]🔗

Validate that min_time < max_time for the case.

Parameters:

  • min_time (float) – The minimum time.

  • max_time (float) – The maximum time.

Raises:

SimpleBenchValueError – The min_time is greater than max_time.

static validate_variation_cols(
variation_cols: dict[str, str] | None,
kwargs_variations: dict[str, list[Any]],
) → dict[str, str][source]🔗

Validate the variation_cols dictionary.

Parameters:

  • variation_cols (dict[str, str] | None) – The variation_cols dictionary to validate or None.

  • kwargs_variations (dict[str, list[Any]]) – The kwargs_variations dictionary to validate against.

Returns:

A shallow copy of the validated variation_cols dictionary or {} if not provided. Each key is a keyword argument name from kwargs_variations, and each value is a non-blank string to be used as the column label for that argument in reports.

Return type:

dict[str, str]

Raises:

  • SimpleBenchTypeError – If the variation_cols is not a dictionary or if any key or value is not a string.

  • SimpleBenchValueError – If any key is not found in kwargs_variations or if any value is a blank string.

property variation_cols: dict[str, str]🔗

Keyword arguments to be used for columns to denote kwarg variations.

Each key is a keyword argument name, and the value is the column label to use for that argument. Only keywords that are also in kwargs_variations can be used here. These fields will be added to the output of reporters that support them as columns of data with the specified labels.

Note that all keys in variation_cols must be present in kwargs_variations and updating it may require changes to both variation_cols and kwargs_variations_cols.

Updating variation_cols does not automatically update kwargs_variations, and vice versa.

Returns:

A dictionary mapping keyword argument names to column labels.

Return type:

dict[str, str]

property warmup_iterations: int🔗

The number of warmup iterations to run before the benchmark.

class simplebench.ImageType(value)[source]🔗

Bases: str, Enum

Enumeration of image types for graph output.

PNG = 'png'🔗

PNG (Portable Network Graphics) image format.

SVG = 'svg'🔗

SVG (Scalable Vector Graphics) image format.

class simplebench.JSONOptions(*, full_data: bool = False)[source]🔗

Bases: ReporterOptions

Class for holding JSON reporter specific options in a Choice or Case.

This class provides additional configuration options specific to the JSON reporter. It is accessed via the options attribute of a Choice or Case instance.

Variables:

full_data (bool) – Whether to include full data in the JSON output.

property full_data: bool🔗

Return whether to include full data in the JSON output.

Returns:

Whether to include full data in the JSON output.

Return type:

bool

class simplebench.Results(
*,
group: str,
title: str,
description: str,
n: int,
rounds: int,
total_elapsed: float,
iterations: Sequence[Iteration],
variation_cols: dict[str, str] | None = None,
variation_marks: dict[str, Any] | None = None,
interval_unit: str = 'ns',
interval_scale: float = 1e-09,
ops_per_interval_unit: str = 'ns',
ops_per_interval_scale: float = 1e-09,
memory_unit: str = 'bytes',
memory_scale: float = 1.0,
ops_per_second: OperationsPerInterval | None = None,
per_round_timings: OperationTimings | None = None,
memory: MemoryUsage | None = None,
peak_memory: PeakMemoryUsage | None = None,
extra_info: dict[str, Any] | None = None,
)[source]🔗

Bases: object

Container for the results of a single benchmark test.

The Results class holds all relevant information about a benchmark test’s execution and its outcomes. It is immutable after creation to ensure data integrity.

Variables:

  • group (str) – The reporting group to which the benchmark case belongs. (read only)

  • title (str) – The name of the benchmark case. (read only)

  • description (str) – A brief description of the benchmark case. (read only)

  • n (int) – The n weighting the benchmark assigned to the iteration for purposes of Big O analysis. (read only)

  • rounds (int) – The number of rounds in the benchmark case. (read only)

  • variation_marks (MappingProxyType[str, Any]) – A dictionary of variation marks used to identify the benchmark variation. (read only)

  • variation_cols (MappingProxyType[str, str]) – The columns to use for labelling kwarg variations in the benchmark. (read only)

  • interval_unit (str) – The unit of measurement for the interval (e.g. “ns”). (read only)

  • interval_scale (float) – The scale factor for the interval (e.g. 1e-9 for nanoseconds). (read only)

  • ops_per_interval_unit (str) – The unit of measurement for operations per interval (e.g. “ops/s”). (read only)

  • ops_per_interval_scale (float) – The scale factor for operations per interval (e.g. 1.0 for ops/s). (read only)

  • memory_unit (str) – The unit of measurement for memory usage (e.g. “bytes”). (read only)

  • memory_scale (float) – The scale factor for memory usage (e.g. 1.0 for bytes). (read only)

  • iterations (tuple[Iteration, ...]) – A tuple of Iteration objects representing each iteration of the benchmark. (read only)

  • ops_per_second (OperationsPerInterval) – Statistics for operations per interval. (read only)

  • per_round_timings (OperationTimings) – Statistics for per-round timings. (read only)

  • memory (MemoryUsage) – Statistics for memory usage. (read only)

  • peak_memory (PeakMemoryUsage) – Statistics for peak memory usage. (read only)

  • total_elapsed (float) – The total elapsed time for the benchmark. (read only)

  • extra_info (MappingProxyType[str, Any]) – Additional information about the benchmark run. This is a read-only property that returns a mapping proxy to prevent external mutation. (read only)

as_dict(
full_data: bool = False,
) → dict[str, Any][source]🔗

Returns the benchmark results and statistics as a JSON-serializable dictionary.

property description: str🔗

A brief description of the benchmark case.

property extra_info: dict[str, Any]🔗

Additional information about the benchmark run.

property group: str🔗

The reporting group to which the benchmark case belongs.

property interval_scale: float🔗

The scale factor for the interval (e.g. 1e-9 for nanoseconds).

property interval_unit: str🔗

The unit of measurement for the interval (e.g. “ns”).

property iterations: tuple[Iteration, ...]🔗

The tuple of Iteration objects representing each iteration of the benchmark.

property memory: MemoryUsage🔗

Statistics for memory usage.

property memory_scale: float🔗

The scale factor for memory usage (e.g. 1.0 for bytes).

property memory_unit: str🔗

The unit of measurement for memory usage (e.g. “bytes”).

property n: int🔗

The number of rounds the benchmark ran per iteration.

property ops_per_interval_scale: float🔗

The scale factor for operations per interval (e.g. 1.0 for ops/s).

property ops_per_interval_unit: str🔗

The unit of measurement for operations per interval (e.g. “ops/s”).

property ops_per_second: OperationsPerInterval🔗

Statistics for operations per interval.

property peak_memory: PeakMemoryUsage🔗

Statistics for peak memory usage.

property per_round_timings: OperationTimings🔗

Statistics for per-round timings.

results_section(
section: Section,
) → Stats[source]🔗

Returns the requested section of the benchmark results.

Parameters:

section (Section) – The section of the results to return. Must be Section.OPS or Section.TIMING.

Returns:

The requested section of the benchmark results.

Return type:

Stats

property rounds: int🔗

The number of rounds the benchmark ran per iteration.

property title: str🔗

The name of the benchmark case.

property total_elapsed: float🔗

The total elapsed time for the benchmark.

property variation_cols: mappingproxy[str, str]🔗

The columns to use for labelling kwarg variations in the benchmark.

property variation_marks: mappingproxy[str, Any]🔗

A dictionary of variation marks used to identify the benchmark variation.

class simplebench.RichTableOptions(virtual_width: int | None = None)[source]🔗

Bases: ReporterOptions

Class for holding Rich table reporter specific options in a Choice.

This class provides additional configuration options specific to the JSON reporter. It is accessed via the options attribute of a Choice instance.

Variables:

virtual_width (int, optional) –

The width of the Rich table output in characters when rendered to the filesystem or via callback. Must be between 80 and 1000 characters or None. If None, no width constraint is applied.

The virtual width is used to determine how the table should be formatted when rendered to non-console outputs, such as files or callbacks. This allows for better control over the appearance of the table in different contexts.

property virtual_width: int | None🔗

Return the virtual width of the Rich table output.

The virtual width is used when rendered to the filesystem or via callback.

Returns:

The virtual width of the Rich table output in characters.

Return type:

int, optional

class simplebench.ScatterPlotOptions(
width: int | None = None,
height: int | None = None,
dpi: int | None = None,
y_starts_at_zero: bool | None = None,
x_labels_rotation: float | None = None,
style: Style | None = None,
theme: Theme | None = None,
image_type: ImageType | None = None,
)[source]🔗

Bases: MatPlotLibOptions

Scatter Plot options.

Defaults are inherited from MatPlotLibOptions:

  • width: int = 1500

  • height: int = 750

  • dpi: int = 150

  • y_starts_at_zero: bool = True

  • x_labels_rotation: float = 45.0

  • style: Style = Style.DARK_BACKGROUND

  • theme: Theme = Theme.Default

  • image_type: ImageType = ImageType.SVG

class simplebench.Session(
*,
cases: Sequence[Case] | None = None,
verbosity: Verbosity = Verbosity.NORMAL,
default_runner: type[SimpleRunner] | None = None,
args_parser: ArgumentParser | None = None,
progress: bool = False,
output_path: Path | None = None,
console: Console | None = None,
)[source]🔗

Bases: object

Container for session related information while running benchmarks.

Variables:

  • args (Namespace) – The command line arguments for the session.

  • cases (Sequence[Case]) – Sequence of benchmark cases for the session.

  • output_path (Path, optional) – The output path for reports.

  • console (Console) – A Rich Console instance for displaying output.

  • verbosity (Verbosity) – Verbosity level for console output (default: Verbosity.NORMAL)

  • default_runner (type[SimpleRunner]) – The default runner class to use for Cases that do not specify a runner. Defaults to SimpleRunner.

  • show_progress (bool) – Whether to show progress bars during execution. Defaults to False.

  • progress (Progress) – Rich Progress instance for displaying progress bars. (read only)

  • tasks (RichProgressTasks) – The ProgressTasks instance for managing progress tasks. (read only)

  • reporter_manager (ReporterManager) – The ReporterManager instance for managing reporters. (read only)

add(case: Case) → None[source]🔗

Add a Case to the Sequence of Cases for this session.

Parameters:

case (Case) – Case to add to the Session

Raises:

SimpleBenchTypeError – If the value is not a Case instance.

add_reporter_flags() → None[source]🔗

Add the command line flags for all registered reporters to the session’s ArgumentParser.

Any conflicts in flag names with already declared ArgumentParser flags will have to be handled by the reporters themselves.

This method should be called before parse_args().

It is placed in its own method so that a user can customize the ArgumentParser before or after adding the reporter flags as needed.

It also allows the user to unregister reporters before adding the reporter flags if they want to omit specific built-in reporters entirely.

Raises:

SimpleBenchArgumentError – If there is a conflict or other error in reporter flag names.

property args: Namespace | None🔗

The command line arguments for the session. This will be None until the parse_args() method has been called.

property args_parser: ArgumentParser🔗

The ArgumentParser instance for the session.

property cases: Sequence[Case]🔗

Sequence of Cases for this session.

property console: Console🔗

The Rich Console instance for displaying output.

property default_runner: type[SimpleRunner] | None🔗

The session scoped default runner class to use for Cases that do not specify a runner.

extend(
cases: Sequence[Case],
) → None[source]🔗

Extend the Sequence of Cases for this session.

Parameters:

cases (Sequence[Case]) – Sequence of Cases to add to the Session

Raises:

SimpleBenchTypeError – If the value is not a Sequence of Cases.

property output_path: Path | None🔗

The output path for reports.

parse_args(
args: Sequence[str] | None = None,
) → None[source]🔗

Parse the command line arguments using the session’s ArgumentParser.

This method parses the command line arguments and stores them in the session’s args property. By default, it parses the arguments from sys.argv. If args is provided, it will parse the arguments from the provided sequence of strings instead.

Parameters:

args (Sequence[str], optional) – A list of command line arguments to parse. If None, the arguments will be taken from sys.argv. Defaults to None.

Raises:

SimpleBenchTypeError – If the args_parser is not set.

property progress: Progress🔗

The Rich Progress instance for displaying progress bars.

report() → None[source]🔗

Generate reports for all benchmark cases in the session.

report_keys() → list[str][source]🔗

Get a list of report keys for all reports to be generated in this session.

This filters the report choices based on the command line arguments that were set and parsed when the session was created and returns a list of report keys for the reports that should be generated.

Returns:

A list of report keys for all reports to be generated in this session.

Return type:

list[str]

property reporter_manager: ReporterManager🔗

Return the ReporterManager instance for managing reporters.

Returns:

The ReporterManager instance for managing reporters.

Return type:

ReporterManager

run() → None[source]🔗

Run all benchmark cases in the session.

property show_progress: bool🔗

Whether to show progress bars during execution.

property tasks: RichProgressTasks🔗

The RichProgressTasks instance for managing progress tasks.

property verbosity: Verbosity🔗

The Verbosity level for this session.

class simplebench.Style(value)[source]🔗

Bases: str, Enum

Enumeration of graph styles.

Note

The styles correspond to those available in Matplotlib 3.10.6

BMH = 'bmh'🔗

Bayesian Methods for Hackers style for graphs.

See Bayesian Methods for Hackers

CLASSIC = 'classic'🔗

Light background style for graphs.

Classic matplotlib plotting style

DARK_BACKGROUND = 'dark_background'🔗

Dark background style for graphs.

Set black background default line colors to white.

FIVETHIRTYEIGHT = 'fivethirtyeight'🔗

FiveThirtyEight style for graphs.

Replicated styles from FiveThirtyEight.com

See dataorigami.net

GGPLOT = 'ggplot'🔗

ggplot style for graphs.

Replicates the style of R’s ggplot library.

See everyhue.me

GRAYSCALE = 'grayscale'🔗

Grayscale style for graphs.

Set all colors to grayscale

Note

strings of float values are interpreted by matplotlib as gray values.

PETTROF10 = 'petroff10'🔗

Petroff10 style for graphs.

Color cycle survey palette from Petroff (2021):

See arxiv.org and github.com

SEABORN_V0_8 = 'seaborn-v0_8'🔗

Base Seaborn style for graphs.

SEABORN_V0_8_BRIGHT = 'seaborn-v0_8-bright'🔗

Seaborn bright style for graphs.

SEABORN_V0_8_COLORBLIND = 'seaborn-v0_8-colorblind'🔗

Seaborn colorblind style for graphs.

SEABORN_V0_8_DARK = 'seaborn-v0_8-dark'🔗

Seaborn dark style for graphs.

SEABORN_V0_8_DARKGRID = 'seaborn-v0_8-darkgrid'🔗

Seaborn darkgrid style for graphs.

SEABORN_V0_8_DARK_PALETTE = 'seaborn-v0_8-dark-palette'🔗

Seaborn dark palette style for graphs.

SEABORN_V0_8_DEEP = 'seaborn-v0_8-deep'🔗

Seaborn deep style for graphs.

SEABORN_V0_8_MUTED = 'seaborn-v0_8-muted'🔗

Seaborn muted style for graphs.

SEABORN_V0_8_NOTEBOOK = 'seaborn-v0_8-notebook'🔗

Seaborn notebook style for graphs.

SEABORN_V0_8_PAPER = 'seaborn-v0_8-paper'🔗

Seaborn paper style for graphs.

SEABORN_V0_8_PASTEL = 'seaborn-v0_8-pastel'🔗

Seaborn pastel style for graphs.

SEABORN_V0_8_POSTER = 'seaborn-v0_8-poster'🔗

Seaborn poster style for graphs.

SEABORN_V0_8_TALK = 'seaborn-v0_8-talk'🔗

Seaborn talk style for graphs.

SEABORN_V0_8_TICKS = 'seaborn-v0_8-ticks'🔗

Seaborn ticks style for graphs.

SEABORN_V0_8_WHITE = 'seaborn-v0_8-white'🔗

Seaborn white style for graphs.

SEABORN_V0_8_WHITEGRID = 'seaborn-v0_8-whitegrid'🔗

Seaborn whitegrid style for graphs.

SOLARIZE_LIGHT2 = 'Solarize_Light2'🔗

Solarized light style for graphs.

Solarized color palette taken from ethanschoonover.com

TABLEAU_COLORBLIND10 = 'tableau-colorblind10'🔗

Tableau colorblind10 style for graphs.

class simplebench.Theme(
rcparams: dict[str, Any] | None = None,
)[source]🔗

Bases: RcParams

An immutable MatPlotLib base theme class for the graphs.

This is a subclass of matplotlib.RcParams that represents a theme for Matplotlib graphs. It can be used to define custom styles for Matplotlib graphs generated by SimpleBench.

See Customizing Matplotlib with style sheets and rcParams for more information on customizing Matplotlib themes.

replace(
rcparams: dict[str, Any],
) → Theme[source]🔗

Replace some parameters in the theme and return a new Theme instance without modifying the current instance.

new_theme = old_theme.replace({'axes.grid': False, 'figure.dpi': 200})
Parameters:

rcparams (dict[str, Any]) – The new rcParams to use for the theme.

Returns:

A new Theme instance with the updated rcParams.

Return type:

Theme

class simplebench.Verbosity(value)[source]🔗

Bases: int, Enum

Verbosity level enums for console output.

Defined levels are:

  • QUIET: Only requested output, errors, warnings and critical messages are shown.

  • NORMAL: Normal messages are shown, including status displays during runs.

  • VERBOSE: All messages are shown and status displays during runs.

  • DEBUG: All messages are shown, including debug messages and status displays during runs.

DEBUG = 5🔗

All messages are shown, including debug messages and status displays during runs.

This is incompatible with quiet.

NORMAL = 1🔗

Normal messages are shown, including status displays during runs.

This is the default verbosity level and is incompatible with quiet.

QUIET = 0🔗

Only requested output, errors, warnings and critical messages are shown. Status displays are not shown during runs.

This is incompatible with all other output levels.

VERBOSE = 2🔗

All messages are shown and status displays during runs.

This is incompatible with quiet.

simplebench.benchmark(
group: str | Callable[[...], Any] = 'default',
/,
*,
title: str | None = None,
description: str | None = None,
iterations: int = 20,
warmup_iterations: int = 10,
rounds: int = 1,
min_time: float = 5.0,
max_time: float = 20.0,
variation_cols: dict[str, str] | None = None,
kwargs_variations: dict[str, list[Any]] | None = None,
options: list[ReporterOptions] | None = None,
n: int = 1,
use_field_for_n: str | None = None,
) → Callable[[Callable[[P], R]], Callable[[P], R]][source]🔗

A decorator to register a function as a benchmark case.

This module uses a global registry to store benchmark cases created via the @benchmark decorator. This enables a streamlined workflow where users simply decorate functions and call main().

Note

Importing a module that uses @benchmark will register its cases globally. For testing, use clear_registered_cases() to reset state between tests.

This simplifies creating a Case by wrapping the decorated function. The decorated function should contain the code to be benchmarked.

It is important to note that the decorated function will be called within the context of a SimpleRunner.run() call, which means it should not handle its own timing or iterations.

The args provided to the decorator are used to create a Case instance, which is then added to a global registry. The original function is returned unmodified, allowing it to be called directly if needed.

The arguments to the decorator are largely the same as those for Case, with the exception of action, which is replaced by the decorated function.

n is included to allow n-weighting the complexity of the benchmark case when using runners that support it.

A minimal example:

from simplebench import benchmark, main


@benchmark
def addition_benchmark():
    '''A simple addition benchmark.'''
    sum(range(1000))


if __name__ == '__main__':
    extra_args = None if len(sys.argv) > 1 else ['--progress', '--rich-table.console']
main(extra_args=extra_args)

You should read the documentation for Case for full details on the parameters and their meanings.

Parameters:

  • group (str, positional-only) – The benchmark reporting group to which the benchmark case belongs for selection and reporting purposes. It is used to categorize and filter benchmark cases. Cannot be blank. The group parameter is positional-only. All other parameters must be passed as keyword arguments. When the decorator is used without parameters, the group defaults to ‘default’. This has special handling to allow the decorator to be used easily without any parameters.

  • title (Optional[str]) – The title of the benchmark case. Uses the function name if None. Cannot be blank.

  • description (Optional[str]) – A description for the case. Uses the function’s docstring if None or ‘(no description)’ if there is no docstring. Cannot be blank.

  • iterations (int) – The minimum number of iterations to run for the benchmark.

  • warmup_iterations (int) – The number of warmup iterations to run before the benchmark.

  • rounds (int) – The number of rounds to run the benchmark within each iteration.

  • min_time (int | float) – The minimum time in seconds to run the benchmark. Must be a positive number.

  • max_time (int | float) – The maximum time in seconds to run the benchmark. Must be a positive number greater than min_time.

  • variation_cols (Optional[dict[str, str]]) – kwargs to be used for cols to denote kwarg variations. Each key is a keyword argument name, and the value is the column label to use for that argument. Only keywords that are also in kwargs_variations can be used here. These fields will be added to the output of reporters that support them as columns of data with the specified labels. If None, an empty dict is used.

  • kwargs_variations (Optional[dict[str, list[Any]]]) – A mapping of keyword argument key names to a list of possible values for that argument. Default is {}. When tests are run, the benchmark will be executed for each combination of the specified keyword argument variations. The action function will be called with a bench parameter that is an instance of the runner and the keyword arguments for the current variation. If None, an empty dict is used.

  • options (Optional[list[ReporterOptions]]) – A list of additional options for the benchmark case. Each option is an instance of ReporterOptions or a subclass of ReporterOptions. Reporter options can be used to customize the output of the benchmark reports for specific reporters. Reporters are responsible for extracting applicable ReporterOptionss from the list of options themselves.

  • n (int) – The ‘n’ weighting of the benchmark case. Must be a positive integer.

  • use_field_for_n (Optional[str]) – If provided, use the value of this field from kwargs_variations to set ‘n’ dynamically for each variation.

Returns:

A decorator that registers the function for benchmarking and returns it unmodified.

Return type:

Callable[[Callable[P, R]], Callable[P, R]]

Raises:
simplebench.main(
benchmark_cases: Sequence[Case] | None = None,
*,
argv: list[str] | None = None,
extra_args: list[str] | None = None,
) → int[source]🔗

Main entry point for running benchmarks via a command-line interface.

This function is responsible for setting up the command-line interface, parsing arguments, and executing the benchmark cases.

@benchmark() decorated cases are automatically included and added to the list of benchmark cases passed to this function.

Usage:

This function serves as the main entry point for running benchmarks.

Parameters:

  • benchmark_cases (Optional[Sequence[Case]]) – A Sequence of SimpleBench.Case instances to be benchmarked.

  • argv (Optional[list[str]]) – A list of command-line arguments to parse. If None, defaults to sys.argv.

  • extra_args (Optional[list[str]]) – Additional command-line arguments to include.

Returns:

An integer exit code.

Return type:

int

simplebench.register_reporter(
cls: type[Reporter],
) → type[Reporter][source]🔗

Class decorator to register a Reporter subclass.

This decorator can be applied to any subclass of Reporter to register it with the system.

Subpackages🔗

Submodules🔗