Full625-frame Eiger HDF5 comparison
Measured November5,2026, Apple M4 Pro. Each pass reads/decompresses the complete5.69GB file, then integrates with the same fixed mask and settings. Repeated passes use the OS file cache; cold physical-device bandwidth is not measured.
| Path | Intensity frames/s | Whole file | With Poisson frames/s | Whole file |
|---|
| h5py stream → Dioptrin | 23.4 | 26.68s | 23.7 | 26.42s |
| Parallel decode → Dioptrin | 182.0 | 3.43s | 156.5 | 3.99s |
| Reusable-buffer decode → Dioptrin | 193.1 | 3.24s | 172.3 | 3.63s |
| Jon: fused decode + CSC | 222.3 | 2.81s | 161.1 | 3.88s |
The current h5py producer limits the stream to about23frames/s, while the parallel paths use much more of the machine. The integration-only gain does not translate directly into full-file throughput: read/decode/float64 conversion alone reaches 358.3frames/s, and those stages precede integration in the staged Dioptrin path. Jon's fused path avoids materializing a whole float64 image. Dioptrin remains float64-weighted; Jon uses float32 coefficients. With errors, this Jon adapter reads a doubled signal/variance matrix.
Primary optimized numbers come from five alternating direct paired passes. Standard h5py and parallel allocation numbers come from three whole-file passes. The CSV retains every measured range and observed process/other-process CPU load; different runs experience ordinary host-load variability.
Protocol
- implementation_commit: 10435d6
- thread_budget: 8 decoding/integration workers, OPENBLAS_NUM_THREADS=1, OMP_NUM_THREADS=1
- settings: 2000 uniform two-theta degree bins, full pixel splitting, solid angle, polarization0.965, same detector mask and geometry weights; all625frames in file order.
- sampling: Three alternating whole-file passes for h5py/parallel/reusable/fused comparison. Five additional alternating direct paired passes of reusable Dioptrin and fused Jon supply primary optimized numbers. All results and host CPU load retained.
- full_validation: Parallel-decoded Dioptrin and Jon outputs checked against all625h5py-derived Dioptrin outputs. Direct paired625-frame Jon/reusable-Dioptrin results checked independently in each error mode. Sampled uncompressed versus compressed decoding also checked.
- setup_excluded: HDF5 chunk-index discovery, calibration/LUT/plan initialization, mmap opening and initial buffer allocation. h5py stream opens its dataset inside each timed pass. Fixed calibration/mask reused for every frame.
- cache: No cache purge. Every workflow is warmed with a full625-frame pass before timing. Includes file-backed accesses, decoding, orientation conversion where needed, integration, normalization and output allocation/materialization. Not cold-device I/O or first-file initialization.
- precision: Jon coefficient storage is float32; Dioptrin is float64. Poisson Jon adapter stacks signal and squared-weight rows; Dioptrin accumulates variance in its integration traversal. Both use identical exported float64 normalization.
- scope: Dioptas remains unchanged. These are measured input-pipeline prototypes, not a new native HDF5 API shipped by Dioptrin.
All results/ranges · Numerical checks · Input/build provenance · Benchmark source
Reproduce run.py followed by run_paired.py, then summarize.py. Large HDF5 inputs, mask and exported LUT stay outside Git at their recorded paths. No Dioptas code was modified.