LZ4
By Yann Collet
LZ4 is a lossless data compression algorithm and file format designed to prioritize compression and decompression speed over maximum compression ratio. It belongs to the LZ77 family of dictionary-based compressors and is engineered so that…
Definition
LZ4 is a lossless data compression algorithm and file format designed to prioritize compression and decompression speed over maximum compression ratio. It belongs to the LZ77 family of dictionary-based compressors and is engineered so that decompression runs at speeds comparable to a memory copy on modern hardware. LZ4 trades some compression density for that speed, making it suited to situations where throughput or latency matters more than storage size.
Overview
LZ4 was created by Yann Collet, who also later created Zstandard, and was released as an open-source project focused on one clear goal: make compression and decompression fast enough that using compression costs almost nothing in CPU time. Traditional compressors like DEFLATE or bzip2 achieve smaller output but spend meaningfully more CPU cycles doing so; LZ4 instead accepts a larger compressed size in exchange for decompression speeds that can exceed several gigabytes per second on a single core. This makes LZ4 well suited to systems where compression is applied constantly in a hot path rather than as a one-time archival step. Mechanically, LZ4 works like other LZ77-derived algorithms: it scans input for repeated byte sequences and replaces them with back-references (an offset and a length) into previously seen data, combined with literal bytes for content that has not appeared before. What differentiates LZ4 is a deliberately simple encoding format and a minimal, branch-light decoder loop, avoiding the entropy coding stage (such as Huffman coding) that slower compressors use to squeeze out extra ratio. The format also defines a block-based framing layer and an optional streaming mode, and a related variant, LZ4HC, spends more time during compression to find better matches while keeping the same fast decoder. Among its neighbors, LZ4 sits at the speed end of the compression spectrum. Zstandard, by the same author, generally offers a better balance of ratio and speed and has displaced LZ4 in many newer systems, while Snappy from Google occupies similar territory to LZ4 with comparable speed-first design goals. Compared to gzip or bzip2, LZ4 compresses less densely but decompresses many times faster, which is the opposite trade-off those tools make. This positions LZ4 as the choice when compression time and decompression latency are the binding constraint, not final file size. In practice, LZ4 is embedded inside filesystems (such as ZFS and Btrfs, as a transparent compression option), databases and storage engines (including RocksDB and Apache Kafka) to shrink data on disk or over the network with negligible CPU cost, and in-memory data stores that compress values before caching them. Real-time systems, game engines, and network protocols that need to shave transfer size without adding noticeable latency also commonly choose it. Its small, dependency-free C implementation makes it easy to embed in constrained environments. The main limitation of LZ4 is its compression ratio, which is noticeably lower than DEFLATE, bzip2, or Zstandard at comparable settings, so it is a poor choice when minimizing storage or bandwidth is the priority and CPU time is available to spend. It also lacks built-in strong error detection or encryption, so applications must add checksums separately if data integrity verification is needed. When maximum compression is the goal, tools like xz or Zstandard at high levels are more appropriate; LZ4 remains the right tool specifically when speed dominates the decision.
Key Features
- Prioritizes compression and decompression speed over compression ratio
- Decompresses at speeds comparable to a raw memory copy
- Based on the LZ77 dictionary-based compression family
- Skips entropy coding to keep the decoder loop simple and fast
- Includes an LZ4HC variant for better ratio at slower compression speed
- Supports a block-based frame format and optional streaming mode
- Implemented as a small, dependency-free C library, easy to embed
- Used transparently inside filesystems, databases, and messaging systems