Making LZ4's file compression actually use your cores

How a serial read loop capped a 32-core machine at 2 GB/s, and what fixing it looks like — measured, adversarially tested, and benchmarked against pzstd and zstd.

The problem

LZ4 v1.10.0 added multithreaded file compression (lz4 -T#). It scales nicely up to about 8 threads — and then stops. On a 32-core Xeon compressing a 1.7 GB file from RAM, 16 threads were no faster than 8, and 32 threads were slightly slower:

before the fixafter the fix

-T1 before

415

-T1 after

413

-T8 before

2,047

-T8 after

2,995

-T16 before

1,972

-T16 after

3,675

-T32 before

1,902

-T32 after

3,675

Compression throughput in MB/s: lz4 -1 -T# -c silesia8.tar > /dev/null, 1.7 GB (silesia×8) in tmpfs, best of 3, 2×Xeon 16C/32T. Ranges across repeated sessions: T16 3,515–3,675; T32 3,594–3,675.

Finding the bottleneck

A perf profile of the 16-thread run gave the game away: the process used only ~4.3 CPU-seconds over 0.85 s of wall time — roughly 5 of 16 workers busy. The compression workers weren't slow; they were starving.

The culprit lived in programs/lz4io.c. The function LZ4IO_readAndProcess() was a job that re-submits itself into the same thread pool as the compression jobs, and every input chunk had to pass through it serially:

BEFORE — one serial job feeds every worker ┌────────────────────────────────────────────────────────┐ │ readAndProcess (one at a time, queued behind workers): │ │ malloc 4 MB → fread 4 MB → XXH32 hash → submit │──▶ worker 1..N │ → re-submit itself to the back of the pool queue │ (mostly idle) └────────────────────────────────────────────────────────┘ ~2 GB/s hard ceiling, no matter how many workers AFTER — workers feed themselves; only the hash stays ordered dispatcher: submits N chunk jobs (window-bounded), touches no data worker k: pread(chunk k, own offset) into pooled buffer ─┐ → HashGate turnstile (XXH32, strict block order) ─┼▶ ordered write → compress chunk ─┘ scales with cores until the sequential XXH32 (~3.7 GB/s) is saturated

Three separate costs were welded into that serial chain: a fresh 4 MB malloc per chunk (page-fault storms), the fread itself, and the frame's XXH32 content checksum — which the LZ4 frame format defines as one sequential hash over the whole input. Chained together and queued behind busy workers, they capped the whole pipeline at ~2 GB/s.

The fix

Two commits in ~/LZ4-ks (a0f751f, then hardening in 8354921), touching only programs/lz4io.c:

Parallel reads. For seekable regular files with independent blocks (the default), each compression job pread()s its own 4 MB chunk at its own offset into a pooled, reused buffer. The serial read chain disappears; reads scale with workers. The chunk range is anchored to the stream's starting offset, and the read loop survives EINTR and short reads.
Checksum off the critical path. The sequential XXH32 runs through an ordered HashGate turnstile: workers hash their chunk in strict block order the moment their turn comes, then go straight back to compressing. For fast levels a 12-worker clamp avoids pointless DRAM contention with the hasher; high-compression levels (-9…-12), which are CPU-bound, keep every requested worker.
Nothing else changes. stdin/pipes, dependent blocks (-BD), the legacy format, Windows, and the default single-threaded mode all keep the original serial path. The compressed output is byte-identical to the pre-fix binary in every mode — verified, not assumed.

Re-benchmark: lz4 vs pzstd vs zstd at 16 and 32 threads

Before the fix, pzstd and zstd won the file-CLI race at high thread counts (~3.3 GB/s vs lz4's ~2). That's now reversed. Same machine, same 1.7 GB tmpfs input, all tools at their fastest level, interleaved best-of-3, output to /dev/null:

lz4 (fixed)pzstdzstd -T#

16 threads

lz4 -1 -T16

3,594–3,675 MB/s

pzstd -1 -p16

2,695–2,741

zstd -1 -T16

2,488–2,608

32 threads

lz4 -1 -T32

3,594–3,675 MB/s

pzstd -1 -p32

2,788–2,888

zstd -1 -T32

2,651–2,940

Ranges span two interleaved measurement rounds in the same session. lz4 leads by 22–44% at equal thread counts. (Compression ratios are unchanged and format-inherent: lz4 2.10 vs zstd/pzstd ~2.9.)

Configuration	Before fix	After fix	pzstd	zstd
16 threads, MB/s	1,972	3,594–3,675	2,695–2,741	2,488–2,608
32 threads, MB/s	1,902	3,594–3,675	2,788–2,888	2,651–2,940
HC level `-9 -T32`, MB/s	532 (ref)	552	—	—

Where the new ceiling is — and why 4 GB/s is the honest limit

The LZ4 frame format's default content checksum is one sequential XXH32 stream over the entire input. On this machine XXH32 tops out at ~4.3–4.5 GB/s in isolation and ~3.7 GB/s under real pipeline load (DRAM contention with 12+ compressing cores). Ten alternative architectures were built and measured — dedicated hasher threads, ordered combiners, whole-file maps, NUMA binding, prefetch and throttle schemes — and all converged on the same ~3.7 GB/s wall with the hasher at ~100% duty cycle.

Two things prove the analysis: with --no-frame-crc the same binary reaches 6.4–8.1 GB/s (so nothing else limits scaling), and the only ways past the wall — disabling the checksum by default, splitting into multiple frames, skipping work — all change the output bytes and were rejected as benchmark cheating. The delivered result keeps the output byte-identical and takes everything the format allows.

How it was verified

Byte-identity oracle: every output compared against the pre-fix reference binary — 3 reference corpus md5s, a 712-case adversarial matrix (boundary sizes from empty to 5 GB, every block mode, dictionaries, legacy format, seekable stdin, multi-file mode), and a 105-case post-fix spot matrix. All identical.
Adversarial breaker campaign: truncate-during-compression and append-during-compression races, files whose stat size lies (/proc), 8 concurrent instances on one file, fd and memory exhaustion, SIGINT. It caught one real crash (SIGBUS when a file shrinks mid-read via the original mmap path).
Independent read-only review (separate model, instructed not to invent problems) confirmed 4 more defects: silent wrong output for pre-seeked redirected stdin, a 35–41% HC regression from the worker clamp, fatal EINTR handling, and a non-POSIX MAP_ANONYMOUS gate.
All five defects fixed at the root in commit 8354921 — notably the mmap read path was deleted outright in favor of pread, turning the truncation race into a clean Read error exit. A re-review confirmed the fixes; the one flagged residual (--content-size with pre-seeked stdin) was tested empirically and does not reproduce — behavior is identical to the reference.
Sanitizers & tests: ASan, UBSan, and TSan clean across the matrices and 10× race repros; make -C tests test-lz4-basic passes; every produced file passes lz4 -t and round-trips to a matching md5.

Reproduce it

cd ~/LZ4-ks && ./build-optimized.sh          # PGO+LTO+native binary
cat ~/LZ4-ks-bench/silesia.tar{,,,,,,,} > /dev/shm/silesia8.tar
cd /dev/shm
for T in 1 8 16 32; do /usr/bin/time -f "lz4  -T$T  %e s" ~/LZ4-ks/lz4 -1 -T$T -c silesia8.tar > /dev/null; done
for T in 16 32;     do /usr/bin/time -f "pzstd -p$T %e s" pzstd -1 -p$T -c silesia8.tar > /dev/null; done
for T in 16 32;     do /usr/bin/time -f "zstd  -T$T  %e s" zstd  -1 -T$T -c silesia8.tar > /dev/null; done

Measurements: 2×Intel Xeon 2.80 GHz (16 physical cores, 32 hardware threads), input and output in tmpfs, idle machine, best-of-3 interleaved runs. Code: ~/LZ4-ks commits a0f751f (parallel chunk reads) and 8354921 (five audited fixes); full measurement record in ~/LZ4-ks/OPTIMIZATION-RESULTS.md; audit trails in ~/LZ4-ks-adv2/BREAKER-REPORT.md and ~/LZ4-ks-bench/mtscale/ (DEV-REPORT.md, REVIEW-gpt56sol.md). pigz was excluded here because it was already far behind (~1.3 GB/s at 16–32 threads) in the earlier round.