simplebench.runners modulešŸ”—

Test runners for benchmarking.

class simplebench.runners.SimpleRunner(
*,
case: Case,
kwargs: dict[str, Any],
session: Session | None = None,
runner: Callable[..., Any] | None = None,
)[source]šŸ”—

Bases: object

A class to run benchmarks for various actions.

Parameters:

  • case (Case) ā€“ The benchmark case to run.

  • kwargs (dict[str, Any]) ā€“ The keyword arguments for the benchmark case.

  • session (Session, optional) ā€“ The session in which the benchmark is run.

  • runner (Callable[..., Any], optional) ā€“ The function to use to run the benchmark. If None, uses default_runner() from SimpleRunner.

Variables:

  • case (Case) ā€“ The benchmark case to run.

  • kwargs (dict[str, Any]) ā€“ The keyword arguments for the benchmark case.

  • session (Session, optional) ā€“ The session in which the benchmark is run.

  • run (Callable[..., Any]) ā€“ The function to use to run the benchmark.

case: CasešŸ”—

The benchmark Case to run.

default_runner(
*,
n: int,
action: Callable[[...], Any],
setup: Callable[[...], Any] | None = None,
teardown: Callable[[...], Any] | None = None,
kwargs: dict[str, Any] | None = None,
) → Results[source]šŸ”—

Run a generic benchmark using the specified action and test data for rounds.

This function will execute the benchmark for the given action and collect the results. It is designed for macro-benchmarks (i.e., benchmarks that measure the performance of a function over multiple iterations) where the overhead of the function call is not significant compared with the work done inside the function.

Micro-benchmarks (i.e., benchmarks that measure the performance of a fast function over a small number of iterations) require more complex handling to account for the overhead of the function call.

Parameters:

  • n (int) ā€“ The O(n) ā€˜nā€™ weight of the benchmark. This is used to calculate a weight for the purpose of O(n) analysis. For example, if the action being benchmarked is a function that sorts a list of length n, then n should be the length of the list. If the action being benchmarked is a function that performs a constant-time operation, then n should be 1.

  • action (Callable[..., Any]) ā€“ The action to benchmark.

  • setup (Callable[..., Any], optional) ā€“ A setup function to run before each iteration.

  • teardown (Callable[..., Any], optional) ā€“ A teardown function to run after each iteration.

  • kwargs (dict[str, Any], optional) ā€“ Keyword arguments to pass to the action.

Returns:

The results of the benchmark.

Return type:

Results

kwargs: dict[str, Any]šŸ”—

The keyword arguments for the benchmark function.

run: Callable[..., Any]šŸ”—

Benchmark runner function. Defaults to SimpleRunner.default_runner().

The runner function must accept the following parameters:

n (int): The number of test rounds that will be run by the action on each iteration. action (Callable[ā€¦, Any]): The function to benchmark. setup (Optional[Callable[ā€¦, Any]]): A setup function to run before each iteration. teardown (Optional[Callable[ā€¦, Any]]): A teardown function to run after each iteration. kwargs (Optional[dict[str, Any]]): Keyword arguments to pass to the function being benchmarked.

session: Session | NonešŸ”—

The session in which the benchmark is run.

property variation_marks: dict[str, Any]šŸ”—

Return the variation marks for the benchmark.

The variation marks are defined by the variation_cols and the current keyworded arguments to the function being benchmarked.

The variation marks identify the specific variations being tested in a run from the kwargs values.

Returns:

The variation marks for the benchmark.

Return type:

dict[str, Any]