Compression transforms Kafka messages into smaller representations, reducing network throughput and retained storage. The preferred architecture compresses complete producer batches before transmission; Kafka stores and forwards the compressed data, while consumers decompress it automatically. section 4.4
Kafka supports two placement strategies with different operational consequences:
- Producer-level compression uses gzip, snappy, lz4, or zstd. Setting compression.type to none disables compression. section 4.4
- Batch compression improves ratios because messages in a batch often share similar structure. section 4.4
- Broker-level compression can retain the producer codec or recompress with another codec; changing codecs forces decompression and recompression, adding work. section 4.4
- Compression choices affect CPU usage, message limits, batching, buffer sizing, throughput, storage, and cloud cost, especially for repetitive JSON or Avro payloads. section 10.1.4
A producer-side compression path
A producer groups similar events into a batch, compresses the batch with zstd, and sends it to Kafka. The broker retains the compressed form and forwards it to consumers, which decompress it without special consumer configuration. Keeping the producer codec avoids broker-side recompression. section 4.4
Figure 4.22 Compressing the data on producer side
Trace the producer-side flow from batching and compression through broker storage and consumer delivery. section 4.4
