UP | HOME

Batch kernels pick one instruction set at run time and keep a scalar fallback. A build does not crash on a CPU that lacks a wider set. Nothing in the build asks for -march=native or a target-cpu.

Paths

Path Lanes When it is usable
scalar 1 always
avx 4 f64 x86_64 with AVX
avx2 4, integer kernels x86_64 with AVX2 and FMA
avx512 8 f64 x86_64 with AVX-512F and AVX-512DQ, and rustc 1.89 or newer
neon 2 f64 aarch64, where NEON is baseline

Unset, the widest usable path is chosen: AVX-512, then AVX2, then AVX, then NEON, then scalar. AVX2 is a candidate only when FMA is present too, so a Sandy Bridge machine stays on AVX. FMA is a selection gate. The kernels do not enable it.

The float batches (dist2_many, dist2_pairs, wrap_many, dist2_ortho_diffs) use the AVX kernel for both the avx and avx2 paths. The orthorhombic wrap_many stays on that four-wide kernel even when AVX-512 is selected, because a 64-byte store splits a cache line on most alignments. Orthorhombic dist2_pairs and dist2_many use the full width of the selected path. A lane with |d| / L below one million stays in the vector form when it is more than one box away. A larger quotient, or a NaN, runs the scalar code for that group. The fixed and fixed32 batches use AVX2 or AVX-512. An avx selection runs those on the scalar path.

NEON processes two rows at a time, in the same order and with the same multiply and add as the scalar loop. The odd tail is scalar. dist2_ortho_diffs runs two of those pairs side by side. After abs the quotient is non-negative, so each pair truncates q + (1/2 - eps) and sums (x^2 + y^2) + z^2. A lane at 2^52, or a NaN, sends that group through the scalar loop.

Selection

simd_path() returns the name of the path the next batch will use. Each batch entry reads that choice once, before the row loop.

MINIMAGE_SIMD may be scalar, avx, avx2, avx512, or neon. The variable is read once and cached for the process. An unknown name is ignored. A name this CPU cannot run selects scalar. Tests can override the choice on the calling thread. A worker thread does not see that override, so a batch that fans out must read the path on the calling thread and pass it down.

Packed rows

An orthorhombic packed kernel used to send a whole vector group back to scalar when any lane was a full box away. On the review batch that is most groups, so the wide path paid for the transpose and then ran the scalar loop. The default path was the wide one, and it was slower than scalar for dist2_pairs and wrap_many.

The batch is the orthorhombic cell 10 by 11 by 12, n = 100000. Row i uses t = i * 0.17 and u = i * 0.13 + 1:

  • ps[i] = [sin(t) * 9, cos(t) * 8, sin(t * 0.3) * 7]
  • qs[i] = [cos(u) * 9, sin(u) * 8, cos(u * 0.2) * 7]

A row is far when the absolute difference on some axis is at least the box length. For dist2_pairs that is 55.5% of rows, 68.9% of groups of four, and 86.2% of groups of eight. For dist2_many, with p = ps[0], 37.8% of rows are far. wrap_many sees the same pair differences.

Bytes moved by one call: pairs 5.6 MB (two position arrays and the f64 output), many 3.2 MB, wrap 4.8 MB, and the SoA dist2_ortho_diffs 3.2 MB. At the old scalar pairs time of 0.675 ms, 5.6 MB is 8.3 GB/s. One core's L2 is 2 MiB (16 MiB across eight cores), so 5.6 MB misses that L2. The 320 MiB L3 holds it. On the same machine the SoA kernel moves 3.2 MB in 0.090 ms, about 36 GB/s, and runs 3.5 times faster than its scalar loop. The packed kernels were not waiting on memory.

The 256-bit path lost on the same batches: pairs 0.759 ms against scalar 0.675, wrap 0.646 against 0.564. That is the width that wins on dist2_ortho_diffs. A 512-bit frequency licence does not account for a loss the 256-bit kernel shares. Eight-wide was the slowest of the wide paths because 86% of its groups contained a far row and were thrown away.

Scaling the coordinates by 0.4 keeps every difference inside one box, so no group bails. One process, five rounds, median, n = 100000, milliseconds: pairs scalar 0.181, AVX 0.161, AVX-512 0.172; many scalar 0.179, AVX 0.099, AVX-512 0.087; wrap scalar 0.181, AVX 0.141, AVX-512 0.141. The AoS shuffle is faster than scalar when the vector result is stored. The AVX-512 wrap figure is the four-wide kernel.

The vector form now keeps those lanes. While |d| / L is below one million, the square is abs(d) - L * round(abs(d) / L) and the wrap is the floor form, then +0.0 so -0.0 comes back as +0.0. Both match the scalar bits. Past that quotient the group still runs scalar. MINIMAGE_SIMD still forces a path for tests. With the variable unset, simd_path() is the native path. On this CPU that is AVX-512, which is the default column below.

The machine is a KVM guest, Intel Xeon family 6 model 207, eight cores, AVX-512F and AVX-512DQ. Times are medians of five process medians. Each process runs five interleaved rounds. The range is the minimum and maximum of those five medians. Branch and main processes alternate. Release build, LTO, one codegen unit. Milliseconds per call, n = 100000.

kernel scalar avx avx2 avx512 default main
dist2_pairs 0.6665 (0.6648–0.6675) 0.1630 (0.1603–0.1702) 0.1603 (0.1588–0.1692) 0.1531 (0.1504–0.1590) 0.1558 (0.1525–0.1605) 0.8238 (0.8165–0.8265)
dist2_many 0.5237 (0.5230–0.5247) 0.1384 (0.1380–0.1420) 0.1400 (0.1386–0.1419) 0.0932 (0.0914–0.0989) 0.0933 (0.0915–0.0993) 0.5644 (0.5472–0.5719)
wrap_many 0.5488 (0.5477–0.5540) 0.1784 (0.1774–0.1860) 0.1842 (0.1771–0.1870) 0.1774 (0.1764–0.1844) 0.1786 (0.1779–0.1830) 0.6460 (0.6405–0.6532)
dist2_ortho_diffs 0.3145 (0.3138–0.3164) 0.0852 (0.0848–0.0889) 0.0851 (0.0844–0.0888) 0.0851 (0.0844–0.0888) 0.0853 (0.0844–0.0889) 0.0816 (0.0803–0.0825)

default is auto-dispatch. It matches avx512 on this CPU. All four kernels beat scalar. main still bails the group, so its packed kernels stay slower than scalar. Its dist2_ortho_diffs is a few percent faster than the column here. That kernel already won, and the shared rounding is unchanged.

At n = 1000 the same ordering holds. default pairs is 0.0008 ms against scalar 0.0065.

The arm64 runner is ubuntu-24.04-arm. One process, five rounds, release build. cpu_path_native is neon, so default is that path. These are runner timings. The number is the middle round, and the range is the minimum and maximum of the five rounds. Milliseconds per call, n = 100000.

kernel scalar neon default
dist2_pairs 1.1583 (1.1572–1.1815) 0.4809 (0.4783–0.4826) 0.4743 (0.4722–0.4788)
dist2_many 0.7044 (0.7031–0.7284) 0.5002 (0.4996–0.5011) 0.5026 (0.5010–0.5028)
wrap_many 1.2679 (1.2624–1.3400) 0.7018 (0.7001–0.7039) 0.7004 (0.6997–0.7028)
dist2_ortho_diffs 0.3120 (0.3113–0.3129) 0.2976 (0.2972–0.2986) 0.2984 (0.2975–0.2990)

Pairs, many, and wrap are about 1.4× to 2.4× scalar on this runner. dist2_ortho_diffs is 0.2984 ms against scalar 0.3120 ms. At n = 1000 the default medians are 0.0047, 0.0050, 0.0069, and 0.0031 ms for those four kernels, against scalar 0.0107, 0.0070, 0.0120, and 0.0031.

Bit identity

The same input gives the same output bits on every path, including the sign of zero. Reductions add in the scalar order, (x^2 + y^2) + z^2, with separate multiply and add. There is no fused multiply-add in a value that is stored.

floor_fast and round_away use one truncation on every architecture, the one baseline x86-64 already used. floor_fast(-0.0) is +0.0, so the fractional part of -0.0 is -0.0. f64::floor and f64::round keep the sign of a zero and are not used. x86_64 results of these two functions are unchanged. On aarch64 they used to follow libm, which kept the minus sign.

linkcell's NEON fold still rounds with vrndmq_f64, which is f64::floor and returns -0.0 for -0.0. That fold has to follow this truncation, and fold_edges_match_scalar_on_every_path has to expect the x86_64 sign.

tests/golden/isa-hashes.toml records the SHA-256 of the little-endian output bytes for one input recipe. Every usable path must match the scalar digest. A path the runner cannot execute is not treated as a pass.

The lane layout follows the cluster-pair approach of Páll and Hess, Comput. Phys. Commun. 184, 2641 (2013), https://doi.org/10.1016/j.cpc.2013.06.003. Per-architecture checks of the same bits are the EESSI model of Dröge et al., Softw. Pract. Exp. 53, 176 (2023), https://doi.org/10.1002/spe.3075.