simplebench.stats package

Stats module for SimpleBench benchmarking framework.

class simplebench.stats.MemoryUsage(

*,

iterations: Sequence[Iteration] | None = None,

unit: str = 'bytes',

scale: float = 1.0,

rounds: int = 1,

data: Sequence[int | float] | None = None,

)

Bases: Stats

Container for the memory usage statistics of a benchmark.

Variables:

• unit (str) – The unit of measurement for the memory usage (e.g., “MB”).

• scale (float) – The scale factor for the memory usage (e.g., “1e6” for megabytes).

• rounds (int) – The number of data points in the benchmark.

• data (tuple[float | int, ...]) – Tuple of memory usage data points.

• mean (float) – The mean memory usage.

• median (float) – The median memory usage.

• minimum (float) – The minimum memory usage.

• maximum (float) – The maximum memory usage.

• standard_deviation (float) – The standard deviation of the memory usage.

• relative_standard_deviation (float) – The relative standard deviation of the memory usage.

• percentiles (dict[int, float]) – Percentiles of memory usage.

class simplebench.stats.MemoryUsageSummary(

*,

unit: str,

scale: float,

rounds: int,

mean: float,

median: float,

minimum: float,

maximum: float,

standard_deviation: float,

relative_standard_deviation: float,

percentiles: tuple[float, ...],

)

Bases: StatsSummary

Container for summary statistics of a MemoryUsage benchmark.

This class is exclusive of raw data points.

Variables:

• unit (str) – The unit of measurement for the benchmark (e.g., “ops/s”).

• scale (float) – The scale factor for the interval (e.g. 1 for seconds).

• rounds (int) – The number of data points in the benchmark.

• data (tuple[int | float, ...]) – Always an empty tuple as a StatsSummary object does not contain raw data points.

• mean (float) – The mean operations per time interval.

• median (float) – The median operations per time interval.

• minimum (float) – The minimum operations per time interval.

• maximum (float) – The maximum operations per time interval.

• standard_deviation (float) – The standard deviation of operations per time interval.

• relative_standard_deviation (float) – The relative standard deviation of ops per time interval.

• percentiles (tuple[float, ...]) – Percentiles of operations per time interval.

class simplebench.stats.OperationTimings(

*,

iterations: Sequence[Iteration] | None = None,

unit: str = 'ns',

scale: float = 1e-09,

rounds: int = 1,

data: Sequence[int | float] | None = None,

)

Bases: Stats

Container for the operation timing statistics of a benchmark.

Variables:

• unit (str) – The unit of measurement for the timings (e.g., “ns”).

• scale (float) – The scale factor for the timings (e.g., “1e-9” for nanoseconds).

• rounds (int) – The number of data points in the benchmark.

• data (tuple[int | float, ...]) – Tuple of timing data points.

• mean (float) – The mean time per operation.

• median (float) – The median time per operation.

• minimum (float) – The minimum time per operation.

• maximum (float) – The maximum time per operation.

• standard_deviation (float) – The standard deviation of the time per operation.

• relative_standard_deviation (float) – The relative standard deviation of the time per operation.

• percentiles (dict[int, float]) – Percentiles of time per operation.

class simplebench.stats.OperationTimingsSummary(

*,

unit: str,

scale: float,

rounds: int,

mean: float,

median: float,

minimum: float,

maximum: float,

standard_deviation: float,

relative_standard_deviation: float,

percentiles: tuple[float, ...],

)

Bases: StatsSummary

Container for summary of operation timing statistics of a benchmark.

Variables:

• unit (str) – The unit of measurement for the benchmark (e.g., “ops/s”).

• scale (float) – The scale factor for the interval (e.g. 1 for seconds).

• rounds (int) – The number of data points in the benchmark.

• mean (float) – The mean operations per time interval.

• median (float) – The median operations per time interval.

• minimum (float) – The minimum operations per time interval.

• maximum (float) – The maximum operations per time interval.

• standard_deviation (float) – The standard deviation of operations per time interval.

• relative_standard_deviation (float) – The relative standard deviation of ops per time interval.

• percentiles (dict[int, float]) – Percentiles of operations per time interval.

class simplebench.stats.OperationsPerInterval(

*,

iterations: Sequence[Iteration] | None = None,

unit: str = 'Ops/s',

scale: float = 1.0,

rounds: int = 1,

data: Sequence[int | float] | None = None,

)

Bases: Stats

Container for the operations per time interval statistics of a benchmark.

Variables:

• unit (str) – The unit of measurement for the benchmark (e.g., “ops/s”).

• scale (float) – The scale factor for the interval (e.g. 1 for seconds).

• rounds (int) – The number of data points in the benchmark.

• data (Sequence[int | float]) – List of data points.

• mean (float) – The mean operations per time interval.

• median (float) – The median operations per time interval.

• minimum (float) – The minimum operations per time interval.

• maximum (float) – The maximum operations per time interval.

• standard_deviation (float) – The standard deviation of operations per time interval.

• relative_standard_deviation (float) – The relative standard deviation of ops per time interval.

• percentiles (dict[int, float]) – Percentiles of operations per time interval.

class simplebench.stats.OperationsPerIntervalSummary(

*,

unit: str,

scale: float,

rounds: int,

mean: float,

median: float,

minimum: float,

maximum: float,

standard_deviation: float,

relative_standard_deviation: float,

percentiles: tuple[float, ...],

)

Bases: StatsSummary

Summary of OperationsPerInterval statistics.

Variables:

• unit (str) – The unit of measurement for the benchmark (e.g., “ops/s”).

• scale (float) – The scale factor for the interval (e.g. 1 for seconds).

• rounds (int) – The number of data points in the benchmark.

• mean (float) – The mean operations per time interval.

• median (float) – The median operations per time interval.

• minimum (float) – The minimum operations per time interval.

• maximum (float) – The maximum operations per time interval.

• standard_deviation (float) – The standard deviation of operations per time interval.

• relative_standard_deviation (float) – The relative standard deviation of ops per time interval.

• percentiles (dict[int, float]) – Percentiles of operations per time interval.

class simplebench.stats.PeakMemoryUsage(

*,

iterations: Sequence[Iteration] | None = None,

unit: str = 'bytes',

scale: float = 1.0,

rounds: int = 1,

data: Sequence[int | float] | None = None,

)

Bases: Stats

Container for the peak memory usage statistics of a benchmark.

Variables:

• unit (str) – The unit of measurement for the memory usage (e.g., “MB”).

• scale (float) – The scale factor for the memory usage (e.g., “1e6” for megabytes).

• rounds (int) – The number of data points in the benchmark.

• data (tuple[int | float, ...]) – Tuple of peak memory usage data points.

• mean (float) – The mean memory usage.

• median (float) – The median memory usage.

• minimum (float) – The minimum memory usage.

• maximum (float) – The maximum memory usage.

• standard_deviation (float) – The standard deviation of the memory usage.

• relative_standard_deviation (float) – The relative standard deviation of the memory usage.

• percentiles (dict[int, float]) – Percentiles of memory usage.

class simplebench.stats.PeakMemoryUsageSummary(

*,

unit: str,

scale: float,

rounds: int,

mean: float,

median: float,

minimum: float,

maximum: float,

standard_deviation: float,

relative_standard_deviation: float,

percentiles: tuple[float, ...],

)

Bases: StatsSummary

Container for the summary of peak memory usage statistics of a benchmark.

Variables:

• unit (str) – The unit of measurement for the memory usage (e.g., “MB”).

• scale (float) – The scale factor for the memory usage (e.g., “1e6” for megabytes).

• rounds (int) – The number of data points in the benchmark.

• mean (float) – The mean memory usage.

• median (float) – The median memory usage.

• minimum (float) – The minimum memory usage.

• maximum (float) – The maximum memory usage.

• standard_deviation (float) – The standard deviation of the memory usage.

• relative_standard_deviation (float) – The relative standard deviation of the memory usage.

• percentiles (dict[int, float]) – Percentiles of memory usage.

class simplebench.stats.Stats(

*,

unit: str,

scale: float,

data: Sequence[int | float],

rounds: int = 1,

)

Bases: object

Generic container for statistics on a benchmark.

Variables:

• unit (str) – The unit of measurement for the benchmark (e.g., “ops/s”). (read only)

• scale (float) – The scale factor for the interval (e.g. 1 for seconds). (read only)

• rounds (int) – The number of rounds each data point represents. (read only)

• data (tuple[float | int, ...]) – Tuple of data points. (read only)

• mean (float) – The mean operations per time interval. (read only)

• median (float) – The median operations per time interval. (read only)

• minimum (float) – The minimum operations per time interval. (read only)

• maximum (float) – The maximum operations per time interval. (read only)

• standard_deviation (float) – The standard deviation of operations per time interval. (read only)

• adjusted_standard_deviation (float) – The standard deviation adjusted for the number of rounds. (read only)

• relative_standard_deviation (float) – The relative standard deviation of ops per time interval. (read only)

• adjusted_relative_standard_deviation (float) – The adjusted relative standard deviation of ops per time interval. (read only)

• percentiles (tuple[float, ...]) – Percentiles of operations per time interval. (read only)

property adjusted_relative_standard_deviation: float

The adjusted relative standard deviation of the data.

property adjusted_standard_deviation: float

The standard deviation adjusted for the number of rounds.

This metric “un-suppresses” the variability that is reduced by averaging across multiple rounds. It is calculated by multiplying the standard deviation by the square root of the number of rounds.

property as_dict: dict[str, str | float | dict[int, float] | tuple[int | float, ...]]

Returns the statistics and data as a JSON-serializable dictionary.

This includes all the statistics as well as the raw data points.

The data values are scaled according to the scale factor to provide human-readable values using the base unit rather than the scaled unit.

The unit is normalized to its SI base unit representation. (e.g., “ms” becomes “s”)

The dictionary is mutability-safe as all data is either a primitive or a copy.

Returns:

A dictionary containing the statistics and the scaled data points.

property data: tuple[int | float, ...]

The data points.

classmethod from_dict(

data: dict[str, Any],

) → Stats

Construct a Stats object from a dictionary.

Example

stats_dict = {
    "unit": "ops/s",
    "scale": 1,
    "data": [1000, 2000, 1500, 3000, 2500]
}
stats = Stats.from_dict(stats_dict)
print(stats.mean)  # Output: 2000.0

Parameters:

data (dict) – A dictionary containing the stats data. Must contain ‘data’ key with a non-empty sequence of data points consisting of integers or floats.

Returns:

A Stats object constructed from the provided dictionary.

Raises:

• SimpleBenchTypeError – If the data, unit, or scale arguments are of the wrong type.

• SimpleBenchKeyError – If the data dictionary does not contain a ‘unit’ key and no unit argument is provided.

• SimpleBenchValueError – If the data dictionary does not contain a non-empty ‘data’ key with at least one data point, if the scale argument is not greater than zero, or if the unit argument is an empty string

property maximum: float

The maximum of the data.

property mean: float

The mean of the data.

property median: float

The median of the data.

property minimum: float

The minimum of the data.

property percentiles: tuple[float, ...]

Percentiles of the data.

Returns the 0th through 100th percentiles of the data as an immutable tuple.

property relative_standard_deviation: float

The relative standard deviation of the data.

property rounds: int

The number of rounds each data point represents.

property scale: float

The scale of the data.

property standard_deviation: float

The standard deviation of the data.

property stats_summary: StatsSummary

Returns a StatsSummary object created from this Stats object.

Returns:

A StatsSummary object containing the same statistics as this Stats object.

property unit: str

The unit of the data.

class simplebench.stats.StatsSummary(

*,

unit: str,

scale: float,

rounds: int,

mean: float,

median: float,

minimum: float,

maximum: float,

standard_deviation: float,

relative_standard_deviation: float,

percentiles: tuple[float, ...],

)

Bases: object

Container for summary statistics of a benchmark, exclusive of raw data points.

This is a lightweight, data-only version of the Stats class, suitable for serialization or reporting when raw data points are not needed.

Variables:

• unit (str) – The unit of measurement for the benchmark (e.g., “ops/s”). (read only)

• scale (float) – The scale factor for the interval (e.g. 1 for seconds). (read only)

• rounds (int) – The number of rounds each data point represents. (read only)

• mean (float) – The mean operations per time interval. (read only)

• median (float) – The median operations per time interval. (read only)

• minimum (float) – The minimum operations per time interval. (read only)

• maximum (float) – The maximum operations per time interval. (read only)

• standard_deviation (float) – The standard deviation of operations per time interval. (read only)

• relative_standard_deviation (float) – The relative standard deviation of ops per time interval. (read only)

• percentiles (tuple[float, ...]) – Percentiles of operations per time interval. (read only)

• data (tuple[int | float, ...]) – Always an empty tuple as StatsSummary does not contain raw data points. (read only)

property adjusted_relative_standard_deviation: float

The adjusted relative standard deviation of the data.

property adjusted_standard_deviation: float

The standard deviation adjusted for the number of rounds.

property as_dict: dict[str, str | float | dict[int, float] | tuple[int | float, ...]]

Returns the statistics as a JSON-serializable dictionary.

The data values are scaled according to the scale factor to provide human-readable values using the base unit rather than the scaled unit.

The unit is converted to its SI base unit representation. (e.g., “ms” becomes “s”)

This does not include raw data points, only the statistics.

The dictionary is mutability-safe as all data is either a primitive or a copy.

Returns:

A dictionary containing the statistics.

property data: tuple[int | float, ...]

The data points.

This is always an empty tuple as a StatsSummary does not contain raw data points.

classmethod from_dict(

data: dict[str, Any],

) → StatsSummary

Construct a StatsSummary object from a dictionary.

Example

stats_summary_dict = {
    "unit": "ops/s",
    "scale": 1.0,
    "rounds": 1,
    "mean": 2000.0,
    "median": 2000.0,
    "minimum": 1000.0,
    "maximum": 3000.0,
    "standard_deviation": 790.5694150420949,
    "relative_standard_deviation": 39.52847075252201,
    "percentiles": [1000.0, 1300.0, 1600.0, 1900.0, 2200.0,
                    2500.0, 2800.0, 3000.0, 3000.0]
}
stats_summary = StatsSummary.from_dict(stats_summary_dict)
print(stats_summary.mean)  # Output: 2000.0

Parameters:

data (dict) – A dictionary containing the stats data. Must contain ‘data’ key with a non-empty sequence of data points consisting of integers or floats.

Returns:

A StatsSummary object constructed from the provided dictionary.

Raises:

• SimpleBenchTypeError – If the data, unit, or scale arguments are of the wrong type.

• SimpleBenchKeyError – If the data dictionary does not contain a ‘unit’ key and no unit argument is provided.

• SimpleBenchValueError – If the data dictionary does not contain a non-empty ‘data’ key with at least one data point, if the scale argument is not greater than zero, or if the unit argument is an empty string

classmethod from_stats(

stats: Stats,

) → StatsSummary

Construct a new StatsSummary object from a Stats object.

Parameters:

stats (Stats) – The Stats object to derive the summary from.

Returns:

A new StatsSummary object containing the same statistics as the provided Stats object.

Raises:

SimpleBenchTypeError – If the stats argument is not a Stats object.

property maximum: float

The maximum of the data.

property mean: float

The mean of the data.

property median: float

The median of the data.

property minimum: float

The minimum of the data.

property percentiles: tuple[float, ...]

Percentiles of the data.

property relative_standard_deviation: float

The relative standard deviation of the data.

property rounds: int

The number of rounds each data point represents.

property scale: float

The scale of the data.

property standard_deviation: float

The standard deviation of the data.

property unit: str

The unit of the data.

Subpackages

• simplebench.stats.exceptions package
  • Submodules
    • simplebench.stats.exceptions.memory_usage module
    • simplebench.stats.exceptions.operation_timings module
    • simplebench.stats.exceptions.operations_per_interval module
    • simplebench.stats.exceptions.peak_memory_usage module
    • simplebench.stats.exceptions.stats module

Submodules

• simplebench.stats.memory_usage module
  • MemoryUsage
  • MemoryUsageSummary
• simplebench.stats.operation_timings module
  • OperationTimings
  • OperationTimingsSummary
• simplebench.stats.operations_per_interval module
  • OperationsPerInterval
  • OperationsPerIntervalSummary
• simplebench.stats.peak_memory_usage module
  • PeakMemoryUsage
  • PeakMemoryUsageSummary
• simplebench.stats.stats module
  • Stats
    • Stats.adjusted_relative_standard_deviation
    • Stats.adjusted_standard_deviation
    • Stats.as_dict
    • Stats.data
    • Stats.from_dict()
    • Stats.maximum
    • Stats.mean
    • Stats.median
    • Stats.minimum
    • Stats.percentiles
    • Stats.relative_standard_deviation
    • Stats.rounds
    • Stats.scale
    • Stats.standard_deviation
    • Stats.stats_summary
    • Stats.unit
  • StatsSummary
    • StatsSummary.adjusted_relative_standard_deviation
    • StatsSummary.adjusted_standard_deviation
    • StatsSummary.as_dict
    • StatsSummary.data
    • StatsSummary.from_dict()
    • StatsSummary.from_stats()
    • StatsSummary.maximum
    • StatsSummary.mean
    • StatsSummary.median
    • StatsSummary.minimum
    • StatsSummary.percentiles
    • StatsSummary.relative_standard_deviation
    • StatsSummary.rounds
    • StatsSummary.scale
    • StatsSummary.standard_deviation
    • StatsSummary.unit