Benchmarks

Every number on this page is measured, never asserted. The tables below are rendered at docs-build time straight from the committed baseline JSON (benchmarks/results/baseline-<host>.json) — the docs cannot show a figure the suite did not produce. Each scenario states its hypothesis and its measurement method in its own source file (benchmarks/scenarios/), so the methodology is auditable from the code alone, and every caveat and skip is printed verbatim in the notes under each table.

Where a comparator (ray, accelerate) is not installed on the baseline machine, its row says so and its setup code is counted, not run, from the cited fixtures in benchmarks/scenarios/snippets/ — an honest line count, never a fabricated timing. Some figures here are small, zero, or negative: that is the suite working as intended. On a tiny CPU model, process-startup dominates wall-clock and a checkpoint write or a 40-step recompute falls below the run-to-run noise floor — the notes say so, and the size-independent guarantees (e.g. steps_not_recomputed) are reported alongside. The value shows up at real model scale; the honesty shows up here.

The suite is split into two sections, each rendered as its own table below. Performance measures the overheads a user pays for adopting flashruntime — launch overhead, per-checkpoint cost, submit latency, fan-out throughput, adoption line count — as wall-clock medians against a bare-torchrun or plain sequential baseline. Resilience measures the fault-tolerance guarantees themselves, under real failure injection rather than modelling:

• Correct classification — fault_recovery_matrix runs five distinct fault types (import error, mid-run SystemExit, a worker killed mid-run by an external SIGKILL, a SIGKILL inside the checkpoint-write window, and a corrupted newest part) and counts — from terminal run state, never asserts — how many the typed recovery taxonomy routes to the right action.
• Integrity under kill -9 — checkpoint_integrity fires repeated SIGKILLs that land inside the checkpoint write window; the parts-first / manifest-last commit means a torn write is never a valid checkpoint, so resume falls back to the last verified step. The naive torch.save comparator is killed in the same window and its corruption rate is reported beside ours.
• Goodput under a storm — crash_storm fans out a batch of trials with half of them armed to crash on their first attempt, then measures the fraction of useful work retained (and the tail actually recomputed) once every crashed trial auto-resumes from its checkpoint, with zero manual intervention.
• Measured MTTD/MTTR — lease_recovery_latency boots the real FastAPI coordinator over real loopback sockets, kills a worker mid-lease, and times detection (lease-expiry sweep) and recovery (re-claim → artifact commit) end to end, alongside steady-state claim/heartbeat round-trip latency.

Each resilience number is COUNTED or timed from observable run state, never baked into an assertion. The failure injection lives in benchmarks/faults.py (crashy-trainer generation, kill -9 timing, part corruption), each scenario's measurement method is stated in its own source file under benchmarks/scenarios/, and the long chaos loops carry a bench_stress marker so pytest -m bench_stress re-runs them.

Reproduce the whole baseline yourself:

python -m benchmarks run --all --repeats 5

Run a single scenario, or a fast labelled smoke:

python -m benchmarks run --scenario recovery_economics
python -m benchmarks run --all --smoke

Measured on:

Reproduce every number below with:

python -m benchmarks run --all --repeats 5

Summary

Performance

Resilience

adoption_cost

Hypothesis: Adopting flashruntime is a handful of lines and a tiny dependency footprint — less code and a faster import than ray or accelerate.

Measured: 7 lines to adopt (p10 7, p90 7, 20 repeats)

• adoption LOC = inserted/changed non-blank lines from a vanilla script to its framework-ready form (difflib); snippets are cited from each project's own docs
• LOC is deterministic — repeats do not vary it (import time is the median of 5 subprocess timings)
• ray not installed — import-time/dep-count comparator skipped
• accelerate not installed — import-time/dep-count comparator skipped

fanout_throughput

Hypothesis: A local flash.submit fan-out runs at a tasks/minute rate close to a plain sequential subprocess loop (fan-out is sequential by design), and its per-task overhead amortizes away as task duration grows.

Measured: 111 tasks/min (p10 111, p90 111, 16 repeats)

• throughput = tasks / wall × 60, MEASURED over a single 16-task 0.5 s fan-out (p10==p90==median — one sweep, no repeat spread; the brief runs the full scenario once)
• overhead_frac = (flash_wall − ideal_serial_work) / flash_wall, where ideal_serial_work = count × duration = Σ INTENDED sleeps and NOTHING else — it ignores ALL process overhead (python startup, imports, spawn, collection), so the fraction deliberately CHARGES python startup to overhead. That is the honest 'ideal' floor; it never flatters flashruntime
• amortization curve counts: 0.5 s×16, 2 s×6, 5 s×4 — longer legs use fewer tasks (amortization is per-task, count-independent) to bound wall-clock. The 0.5 s fraction is LARGE (startup is a big slice of 0.5 s) and the 5 s fraction is small; that spread IS the finding, shipped as-is, never trimmed
• local fan-out is SEQUENTIAL by design (each trial's outputs are collected before the next runs), so this measures per-task ORCHESTRATION overhead and its amortization, NOT a parallel speedup — the flash and sequential rates are expected to be close

hpo_sweep

Hypothesis: flashruntime runs an 8-trial sweep with far less setup code than a hand-rolled loop or ray.tune, at comparable local wall-clock.

Measured: 6.12 seconds (p10 5.98, p90 6.26, 20 repeats)

• flash.submit runs trials SEQUENTIALLY locally, so wall-clock ≈ the for-loop baseline; flashruntime's HPO value is orchestration + result collection + fault-tolerance, not local parallelism
• peak RSS is a shared RUSAGE_CHILDREN high-water mark across all trials (largest single child)
• ray.tune wall-clock is NOT measured here (ray not installed — a heavy, throwaway-venv comparator); its setup LOC is counted from the committed, cited snippet ray_tune_hpo.py

launch_overhead

Hypothesis: flash.submit adds under a second of wall-clock over a bare torchrun launch.

Measured: 0.0443 seconds (p10 -0.203, p90 0.24, 20 repeats)

• overhead = flash.submit wall-clock minus identical bare torchrun, paired per repeat
• both launches share the same FLASHML_CKPT_DIR/OUTPUT_DIR isolation (a fresh temp)

loop_overhead

Hypothesis: The ft training loop adds negligible per-step cost; a checkpoint is a few to low-tens of ms.

Measured: -1.25 ms/checkpoint (p10 -13.7, p90 6.19, 20 repeats)

• per-checkpoint = (ckpt=50 run - checkpoint-disabled run) / 4; startup+loop cancel in the delta
• a per-checkpoint figure at or below zero means the write cost is BELOW the run-to-run noise floor for this tiny model (a few-KB state dict) — checkpoints are effectively free here; a larger model would surface a positive cost
• steps/sec and the ratio are startup-dominated at this step count and compare DIFFERENT scripts (ft MLP vs vanilla Linear) — indicative, not identical-work
• checkpoint-every=0 raises ZeroDivisionError in the example; a sentinel > steps disables instead

recovery_economics

Hypothesis: Auto-resume from a checkpoint finishes a crashed run faster than a raw rerun-from-zero, and never recomputes the steps past the last checkpoint.

Measured: 0.259 seconds saved (p10 -0.368, p90 2.44, 20 repeats)

• raw-torchrun cost modelled as t_crash + t_full (no checkpoint ⇒ rerun the whole job)
• auto-resume verified to restart from the step-40 checkpoint (trials[0].resumed_from == 40)
• seconds-saved is small here because torchrun startup (~2 s) dominates and the 40 recomputed steps of this tiny model cost only a fraction of a second; the saving scales with the compute between the last checkpoint and the crash — negligible at smoke size, hours on a real job
• steps_not_recomputed (40) is the size-INDEPENDENT guarantee: resume never re-does work past the last valid checkpoint, whatever a step costs

submit_latency

Hypothesis: flash.submit adds a small, roughly constant overhead before user code hits its first step — dominated by child-process launch, not flashruntime bookkeeping — invisible next to any real training run.

Measured: 0.0249 s (p50 cold) (p10 0.0239, p90 0.0259, 20 repeats)

• total = child.first_step_ts − parent.t0 (submit call → first training step), MEASURED from wall-clock instants the child stamps into metrics.json — never a baked figure
• phase_launch_s (submit call → child proc start) + phase_child_s (proc start → first step) == total by construction; interpreter startup lands in launch (it IS launch cost)
• cold = fresh output_dir per submit (cold dir/page cache); warm = 2nd..Nth of a reused dir (hot cache). The child is a fresh process either way, so cold≈warm — the small delta is the OS file-cache effect, MEASURED, not assumed away
• this is a CONSTANT per-launch overhead: invisible at real-training scale (minutes+); it matters only for very short jobs or very wide fan-outs — which is why we split and report it rather than amortize it away

checkpoint_integrity

Hypothesis: Under repeated kill -9s inside the checkpoint write window, flashruntime's parts-first/manifest-last commit means every resume lands on a hash-verified manifest (integrity_rate → 1.0), while a naive torch.save('latest.pt') overwriting one file in place is truncated by the same kill and fails to reload.

Measured: 1 integrity_rate (p10 1, p90 1, 20 repeats)

• integrity is COUNTED from run state (terminal SUCCEEDED + a hash-verified latest manifest still present + a resume from a verified earlier step>0), never asserted — a mishandled in-window kill lowers the rate as a FINDING, not a test to fix
• integrity_rate = survived_hits / torn_writes_hit — the denominator is IN-WINDOW KILLS ONLY (20/20 iterations); window-missed iterations (0/20 — a clean uninterrupted run whose kill never landed) are EXCLUDED entirely, never counted as trivial 1.0 successes that would inflate the rate
• honesty: the write window is open a LARGE fraction of each step BY DESIGN (checkpoint_every=1, an 8 MB part per step widens the part-on-disk/manifest-absent gap), so hitting it is near-guaranteed — the claim is that flash SURVIVED every hit, NOT that hitting the window was hard
• honesty: torn_writes_hit certifies a kill during the manifest-absent COMMIT window (which includes the part-complete/manifest-pending sub-case), not necessarily a byte-torn part file — the guarantee under test is that a manifest-less step is never restored, however far its parts got
• mean resume step across torn-write hits: 1.0 (a verified EARLIER checkpoint, never the torn one)
• naive comparator: torch.save(state, 'latest.pt') overwritten in place each step, killed mid-write, then torch.load — raised on 20/20 iterations (mid-write kills landed on 20/20); observed failure modes, verbatim exception classes: EOFError×20

crash_storm

Hypothesis: A 16-trial fan-out where every even trial crashes mid-run still completes 16/16 with zero human interventions — flashruntime auto-resumes each crash from its own checkpoint — at a bounded, MEASURED goodput and wall-clock cost.

Measured: 16 completed/16 (p10 16, p90 16, 20 repeats)

• completions, goodput, crashed_first_attempt are COUNTED from run state (len(run.trials) and each trial's metrics steps/resumed_from), never asserted — a storm that completes <16/16 ships as the measured number, a FINDING
• goodput_lower_bound = Σ steps / Σ (steps + resumed_from): a WORST-CASE charge that re-counts each crashed trial's resumed_from AS IF it were recomputed (it was NOT — checkpointing preserved it), making the fraction a pessimistic lower bound that can never flatter flashruntime. The key NAMES the computation; it is not an executed-steps measurement
• recompute_fraction = Σ (crashed_at − resumed_from) / Σ steps, MEASURED from each crashed trial's ACTUAL crash step (faults.py writes crashed_at.json just before the raise; the resumed attempt folds it into metrics): the real redundant work, 0 when the crash fires AT a checkpoint step — the reuse-aware truth the lower bound deliberately over-charges
• manual_interventions = 0.0 is DERIVED: the max_restarts=1 recovery loop auto-resumed every WORKER_CRASH with no human in the loop (a bare torchrun needs one restart per crash)
• local fan-out is SEQUENTIAL by design (each trial's outputs are collected before the next runs), so both sweeps' wall-clock is a sum over trials — this measures fault-tolerant goodput, not throughput

fault_recovery_matrix

Hypothesis: Automated recovery does the right typed thing across a fault matrix — fail-fast on a deterministic bug, resume from the newest valid checkpoint on transient crashes/mid-run kills/mid-write kills/corruption — with zero human interventions where a bare torchrun needs one per fault.

Measured: 5 correct/5 (p10 5, p90 5, 20 repeats)

• (a) import_error → FAIL_JOB fast-stop ✓ (1 attempt, 0 restarts burned)
• (b) systemexit_mid → auto-resume ✓ (resumed_from=4, 2 attempts)
• (c) mid-run external SIGKILL → resume ✓ (fired=True, resumed_from=2)
• (d) mid-write SIGKILL → resume from verified earlier step ✓ (fired=True, resumed_from=1)
• (e) corrupt newest part → resume from earlier valid ✓ (corrupt@8, resumed_from=6)
• correctness is COUNTED from run state (terminal state, attempts count, trials[0].resumed_from), never asserted
• manual_interventions_flash = cases the automation did NOT resolve correctly; torchrun modelled at 5 (no typed classification, no auto-resume — every fault needs a human to notice and resubmit)
• torchrun comparator (case b, bare torchrun argv, nproc=1): crash run exits nonzero with NO retry; recovery modelled as t_crash+t_full=1.87s (mean of 20; no checkpoint ⇒ rerun from zero) — labelled modelled, house convention

lease_recovery_latency

Hypothesis: Mode A detects a dead worker and requeues its task within lease_seconds + 2 s (the sweeper guarantee), and a healthy claim/heartbeat round-trip is a few ms — so recovery time (MTTR) is dominated by the tunable lease window, not coordinator overhead. Measured against the REAL coordinator over real sockets.

Measured: 0.0035 s (MTTR) (p10 0.0035, p90 0.0035, 20 repeats)

• measured against the REAL coordinator (flashruntime.service.app:app in a uvicorn SUBPROCESS on a free loopback port) over real HTTP — TestClient is in-process ASGI and would collapse the round-trip we measure, so it cannot back a latency claim
• first measured slice of the Stage-8 metrics debt (ledger-derived MTTD/MTTR): this exercises the lease loop end-to-end over the wire; the aggregate metrics reuse these events
• MTTD = B-reclaims-task − A-last-heartbeat, MEASURED; mttd_bound_s = lease_seconds (3.0) + sweeper period (2.0) = 5.0 s is the sweeper GUARANTEE — the observed value is faster because B's own claim sweeps first
• MTTD resolution is bounded by B's 50 ms claim-poll cadence (±one poll)
• MTTR = B-completes − B-claims, over the HONEST commit path: PUT the output bytes to /v1alpha1/artifacts/{commit_key} then complete with their real sha256 (server-side sha256-validated); it is ONE death cycle so p10/p90 == median — the N-wide spread is the claim round-trip in the comparators
• lease_seconds=3.0 chosen so the death cycle stays fast; the bound scales with whatever lease window a real deployment picks