Data tiering separates frequently accessed hot data from older cold data. Recent records remain on Kafka broker disks for low-latency access, while older rolled-over segments move to cheaper remote storage, reducing local-disk pressure and storage cost. section 7.1.5 section 1.3.1
A Kafka tiering design combines these operational properties:
- Retains data beyond local-disk capacity, potentially for an indefinite period. section 7.1.5
- Moves older segments and associated metadata to remote storage after rollover. section 7.1.5
- Keeps consumer access transparent: consumers continue normal operations without knowing where records reside. section 7.1.5
- Requires successful remote copying before Kafka cleans up corresponding local logs. section 7.1.5
- Introduces dependency on remote-storage availability when consumers request cold data, while non-remote operations can continue normally. section 7.1.5
Keeping event history without treating local disks as unlimited
An event-driven platform keeps recent events on broker-local disks for fast access and periodically offloads older events to Amazon S3, Azure Blob Storage, Google Cloud Storage, or a custom cloud or on-premises system. This supports longer retention, but does not make Kafka a universal storage solution: loading historical data into microservices on demand can create serious performance bottlenecks. section 7.1.5 section 7.0
Figure 7.13 When the tiered storage feature is enabled, the cold data is moved periodically to cheaper, cloud-based storage.
Use the architecture diagram to contrast low-latency local hot storage with periodically offloaded, cheaper cloud-based cold storage. section 7.1.5
