Step each change added, in points of the baseline value: −13 pts means the metric dropped by 13 % of what it was on the baseline. Everything is a cost, so down is better. An empty cell: the step did not measure that metric.
A step inside the run-to-run range of the two columns it compares, or under 3 points, is noise on this machine and is shown as ≈. Nothing here blocks a change; the page only shows its effect.
Left to right: the baseline, then each step in the order it was recorded; every column includes the ones before it. Above each bar: the value, how many times lower than the baseline, and the step from the previous column.
Each check runs a failure against the real engine on a fresh state file and looks at what is left.
Every check that fails on the last step, and every step that raised a cost beyond noise.
Median of the repeats; the range is in the tooltip of each cell. Green: the value meets the design target of the last row.
Workload. Empty tasks, so what is timed is the orchestration alone: with real tasks the same cost is added on top of the task time. Every run starts from a new state file and a temporary HOME.
Reading it. The numbers are from one machine and move a few percent between runs; compare columns of this page with each other, not with another machine.
Record a step: python -m benchmarks.refactor_optimization.bench --label <step>, add its line to
STEPS in report.py, then python -m benchmarks.refactor_optimization.report.