Making LZ4’s Multithreaded File Compression Scale

What this is about

LZ4 is the codec systems reach for when compression must never become the bottleneck: it sits inside the Linux kernel (zram, zswap, squashfs), ZFS and btrfs, game engines, and databases. Version 1.10.0 of the command-line tool added multithreaded compression (-T#). This post describes making that multithreading actually scale on a many-core machine — up to 2.57× faster than the stock CLI at equal thread counts — plus smaller single-thread gains from a corrected profile-guided build, while keeping the compressed file output byte-for-byte identical to upstream.

The problem

LZ4’s compression core has been micro-optimized by its authors for over a decade; the single-thread hot loop is memory-bandwidth- and branch-limited, and build-level tricks recover only 1–2% there. The opportunity is one level up. Profiling the v1.10.0 CLI at -T16 on a 16-core machine showed only about 5 of 16 cores busy: throughput plateaued near 1.5–2 GB/s no matter how many threads were requested.

The cause is structural. The file-compression pipeline reads the input through a single self-resubmitting job in the same thread pool as the compression workers: allocate a fresh 4 MB buffer, fread the next chunk, run the frame checksum serially, then re-queue behind the compression jobs. Sixteen workers wait on one reader. On top of that, the project’s object-file cache turned out to silently defeat profile-guided optimization in a way that is easy to miss (details below).

LZ4’s reputation rests on the stability of its format, so the binding constraint was: the bytes coming out must not change. Every optimization had to keep file-mode output identical to the stock binary, keep every other mode identical to the pre-change binary, survive adversarial and sanitizer testing, and leave the serial code path untouched for pipes, dependent-block mode, the legacy format, and platforms without pread.

Summary

Final measured results

Protocol: pristine stock lz4 v1.10.0 (upstream commit 0774d05, plain make, its own built-in -T# multithreading) versus the patched build, both compressing the same 1.7 GB corpus (8× silesia.tar in tmpfs) to /dev/null: lz4 -<level> -T<threads> -c silesia8.tar. Runs interleaved stock/patched, best of 3, on an idle 2× Xeon 2.80 GHz box (16 physical cores, 32 hyperthreads). 64 threads means 2× oversubscription.

Reading the table: the advantage is level-dependent for a physical reason. At -1 the stock serial read→checksum→dispatch loop starves the workers (stock stalls at 3.5–3.8× scaling), so removing it is worth 1.88–2.57×. At the high-compression levels the compression itself dominates (30–80 MB/s per core) and both binaries scale to ~19–20× on 16 cores plus hyperthreads; the remaining 7–27% comes from HC-aware PGO, LTO and native codegen plus slightly better read/compute overlap. Oversubscribing to 64 threads collapses neither binary. Single-threaded level -1 is unchanged (0.98× is within noise, confirmed with 5 extra interleaved runs).

Against the other parallel compressors

Fast compression is only interesting if it holds up against the tools people would actually reach for. Same machine, same 1.7 GB tmpfs corpus, output to /dev/null, every output decompressed and md5-verified against the original. Before the parallel-read fix, the lz4 CLI lost this race above 8 threads; now:

Ranges are interleaved best-of-3 spreads from the same session. The trade-off is unchanged and worth stating plainly: lz4’s compression ratio on this corpus is 2.10 versus ~2.8–2.9 for the zstd/gzip family — lz4 buys speed with ratio, by design. pzstd also retains a parallel decompression path that the lz4 CLI does not have; lz4’s single-threaded file decode runs at ~1.7 GB/s.

What was changed

All changes preserve the stock code path where the redesign does not apply. Nothing about the frame format, block splitting, or checksumming changed: file-mode output bytes are identical to stock, and every mode is byte-identical to the pre-change binary.

1. Parallel chunk reads for multithreaded file compression (largest gain)

• Root cause, found with perf: LZ4IO_readAndProcess in programs/lz4io.c was a single self-resubmitting job in the same thread pool as compression — fresh 4 MB malloc, serial fread, serial XXH32, re-queue behind the compression jobs. Only ~5 of 16 cores were ever busy.
• Fix: for seekable regular files with independent blocks (POSIX), each compression job now pread()s its own 4 MB chunk into a pooled buffer (EINTR- and short-read-safe), eliminating the serial read chain entirely.
• The frame content checksum (XXH32) is inherently sequential, so it was moved off the read path into an ordered turnstile (“HashGate”): chunks are hashed in order as they become available, overlapped with compression, instead of gating the reads.
• A 12-worker clamp applies only to fast levels, where additional workers merely contend with the serial hasher for DRAM bandwidth; high-compression levels are exempt and use every requested thread.
• The stock serial path is fully preserved for stdin/pipes, dependent blocks (-BD), the legacy format, Windows, unknown stream positions, and the default no--T mode.

2. Opt-in multithreaded benchmark mode (-T# / --mt)

• The in-memory benchmark (lz4 -b) stays single-threaded by default; passing -T#, --threads=#, or --mt runs the identical per-block compress/decompress work on a thread pool over 4 MB blocks, with the XXH64 round-trip verification still active every iteration. Compressed sizes are byte-identical to single-thread -B7.
• Measured with the official harness (-b1 -i3): 13.9–16.5× compression and 12.7–13.2× decompression speedups on silesia and enwik8 (MT block splitting differs from the whole-file single-thread default by ~0.06% in compressed size). The pool is bypassed (no (T n) suffix printed) when it cannot help: inputs under 1 MB or a single block.

3. A profile-guided build — and the two build defects it exposed

• build-optimized.sh builds the CLI with -O3 -march=native -mtune=native -flto plus two-stage PGO. The interesting part is what went wrong twice on the way there.
• Defect one: PGO silently never applied. The project’s build caches objects under a hash of the compiler flags; the instrumented stage and the optimized stage use different flags, so their object paths differed and gcc found no matching profiles — 12 -Wmissing-profile warnings, and the “PGO” binary was actually LTO-only. Fixed by pinning both stages to one object path; the script now greps the build log and aborts if a single missing-profile warning appears.
• Defect two: PGO training bias. Training only on level-1 runs made gcc optimize the fast path at the high-compression path’s expense: the shipped binary lost ~5–6% to stock at -3/-6/-9 even single-threaded, while a plain-make build of the same sources won at every level — proof the code was fine and the profile was the problem. The script now trains on levels 1, 3 and 9; high-compression levels flipped from −6% to +7% single-threaded, with level -1 unregressed.
• Net single-thread effect of the build work: +7% at -3/-6/-9 compression, up to +5.8% decompression on cache-resident inputs (that session ran under external machine load, so it was measured core-pinned with stock and patched runs interleaved), and level -1 compression within noise of stock (it is memory-bandwidth-limited and upstream -O3 codegen is already close to optimal).

The remaining ceiling, measured

Why stop at ~3.8 GB/s? The LZ4 frame format’s content checksum is a single sequential XXH32 over the whole input, and under pipeline load it runs at about 3.7 GB/s on this machine. Compressing with --no-frame-crc reaches 6.4–8.1 GB/s — which proves nothing else in the pipeline limits scaling, and also why that number is not the headline: dropping the checksum (or emitting multiple frames) changes the output bytes, and was rejected as cheating. 4 GB/s with the default checksum is physically unreachable on this hardware.

How the work was verified

Reproducing the results

The patch series is 15 commits on top of upstream 0774d05, touching programs/lz4io.c, programs/bench.c, the CLI option parsing, and the new build-optimized.sh; every measurement round is recorded in OPTIMIZATION-RESULTS.md in the tree. The series has not yet been published as a public fork, so the protocol below is what an independent run looks like once it is:

# pristine baseline
git clone https://github.com/lz4/lz4 lz4-stock
git -C lz4-stock checkout 0774d05 && make -C lz4-stock -j

# patched build, from a sibling checkout of the patch series
# (PGO trains levels 1, 3 and 9; the script aborts on any missing-profile warning)
( cd lz4-patched && ./build-optimized.sh )

# 1.7 GB corpus in RAM (silesia.tar is the standard Silesia corpus as one tar)
rm -f /dev/shm/silesia8.tar
for i in $(seq 8); do cat silesia.tar >> /dev/shm/silesia8.tar; done

# interleaved level x thread sweep
for L in 1 3 6 9; do for T in 8 16 32 64; do
  /usr/bin/time -f "stock   -$L -T$T: %e s" ./lz4-stock/lz4   -$L -T$T -c /dev/shm/silesia8.tar > /dev/null
  /usr/bin/time -f "patched -$L -T$T: %e s" ./lz4-patched/lz4 -$L -T$T -c /dev/shm/silesia8.tar > /dev/null
done; done

# byte-identity: compressed streams from both binaries must hash the same
./lz4-stock/lz4   -9 -T16 -c /dev/shm/silesia8.tar | md5sum
./lz4-patched/lz4 -9 -T16 -c /dev/shm/silesia8.tar | md5sum

# cross-decode: patched output, decoded by the stock binary, must equal the input
md5sum /dev/shm/silesia8.tar
./lz4-patched/lz4 -9 -T16 -c /dev/shm/silesia8.tar | ./lz4-stock/lz4 -d | md5sum

Absolute throughput depends on core count, memory bandwidth and the sequential-XXH32 ceiling of the machine; the shape of the result — stock plateauing at 3.5–3.8× scaling at level -1 while the patched CLI reaches ~9×, and near-parity scaling at high-compression levels with a single-digit codegen win — is what the patch series reproduces. All measurements so far are from one Linux machine (2× Xeon, 16 physical cores); platforms and modes where the pread parallel-read path does not apply (Windows, pipes, dependent blocks, the legacy format) keep the stock serial path.