The Compression Trade-off

• The three costs of compression: ratio, compress speed, decompress speed - and why they trade off.
• How gzip, LZ4, Zstd, XZ, and Brotli work, and why they land where they do.
• A measured ratio-vs-speed comparison from a reproducible Docker harness.
• A workload-driven framework for choosing a codec and level, with four worked settings.
• Practical details that decide real deployments: dictionaries, block size, threads.
• A benchmarking protocol and full references to the codec specs.

1. Preface

   Preface

   Frames the book's purpose - helping engineers choose a compression codec deliberately rather than by habit - and lays out its scope (gzip, LZ4, Zstandard, XZ, and Brotli), audience, and chapter organization.

2. Chapter 1

   The Compression Problem

   Establishes that every lossless codec involves three competing costs - compression ratio, compression throughput, and decompression throughput - and explains why pushing ratio higher costs compression speed disproportionately at the top of the range.

3. Chapter 2

   Inside the Codecs

   Walks through the shared two-stage design (LZ77-style dictionary matching followed by entropy coding) behind gzip, LZ4, Zstandard, XZ, and Brotli, and shows how each codec's window size and entropy-coding choices produce its position on the trade-off curve.

4. Chapter 3

   The Measured Comparison

   Reports the book's own benchmark of all five codecs and multiple levels on a 10.4 MB real source-code corpus, presenting ratio, compress throughput, and decompress throughput in a table and a log-scale ratio-vs-speed plot, and concludes that Zstandard has largely displaced gzip on the frontier.

5. Chapter 4

   Choosing a Codec

   Converts the measured trade-off into a decision framework built around three questions (hot-path latency, write-once/read-many, and scarce resource), applied via a decision diagram and a table of four representative deployment settings.

6. Chapter 5

   Compression in Practice

   Covers the operational details that determine real-world results beyond codec choice: sweeping levels on your own data, using trained dictionaries for small records, block size and multithreading trade-offs, and matching the codec to whether a workload is read-heavy or write-heavy.

7. Chapter 6

   Conclusion: Measure Your Own Data

   Summarizes the five codecs' relative positions on the trade-off curve and closes with a five-step protocol for benchmarking compression on the reader's own data, plus common benchmarking mistakes to avoid.

• Backend, data, or platform engineers who need to choose a compression codec for a storage format, message bus, or API
• Engineers currently picking a codec 'by habit' who want to make the choice deliberately, with reasons
• Anyone deciding among gzip, LZ4, Zstandard, XZ, or Brotli for log pipelines, columnar data files, RPC payloads, backups, or web responses
• Readers who want to run a reproducible benchmark harness on their own data rather than rely on codec folklore
• Working engineers without a background in information theory who still need to reason about compression trade-offs

The preface states the book assumes a working engineer who moves or stores data, and explicitly notes that no background in information theory is required, since the mechanisms are built up from first principles where they matter.

The benchmark harness (in the book's bench/ directory) compresses one fixed corpus - 10.4 MB of concatenated Python 3.11 standard-library source code - with each of five codecs (gzip/DEFLATE, LZ4, Zstandard, XZ/LZMA, Brotli) across several levels per codec, single-threaded (-T1 where supported). It runs on an AMD Ryzen 7 5700X under Docker Desktop using the debian:12-slim image, dated 2026-06-30, and is reproducible via a documented Docker command. Compression ratio (input size divided by output size), compress throughput, and decompress throughput (both in MB/s, from wall-clock time) are measured directly by the harness, not cited from external sources; the book notes ratio is hardware-independent and fully reproducible while throughput is machine-specific.

Almost every system compresses something: log pipelines, columnar data files, RPC payloads, backups, web responses. The choice of codec is usually made once, by habit, and then quietly taxes the system forever, either in CPU spent compressing and decompressing, or in storage and bandwidth spent on bytes that a better codec would have removed. This book is about making that choice deliberately.