Cold data is older or infrequently accessed data retained on cheaper, higher-latency storage. Tiered storage separates it from frequently accessed hot data to balance retention, performance, and cost. section 7.1.5
A tiered architecture typically:
- Keeps hot data on Kafka broker local disks for low-latency access. section 7.1.5
- Periodically offloads older rolled-over segments and metadata to cheaper remote storage. section 7.1.5
- Preserves normal consumer operations while remote access remains transparent, with meaningful errors when remote storage is unavailable. section 7.1.5
Enterprise retention pattern
A Kafka deployment can retain data beyond local-disk limits by moving older segments to Amazon S3, Azure Blob Storage, Google Cloud Storage, or a custom cloud or on-premises system. section 7.1.5
Figure 7.13 When the tiered storage feature is enabled, the cold data is moved periodically to cheaper, cloud-based storage.
Illustrates periodic movement of cold data from broker-local disks to cheaper cloud storage. section 7.1.5
