Data durability is the ability of stored data to remain preserved and recoverable despite failures or restarts. In Kafka, it forms one part of reliability alongside availability and consistency. section 5.0
Durability asks whether data is stored correctly in perpetuity. Replication supports this goal by maintaining redundancy across brokers, while consistency asks whether the cluster persists data exactly as intended. section 5.0 section 4.2
Kafka reliability mechanisms connect durability to several operational concerns:
- Replication distributes data across brokers, supporting durability and availability when individual servers fail. section 5.0
- Acknowledgments help verify that sent messages arrive, while idempotent writes and transactions protect integrity across writes. section 5.0
- Distributed designs must treat failures as expected operational conditions rather than assuming that servers remain reliable. section 4.2
A broker failure
Imagine a Kafka cluster with three brokers. If one broker suddenly fails, durability asks whether the topic data remains preserved elsewhere; replication provides the redundancy needed to address that question. section 5.0
Figure 4.3 Scaling vertically means adding more resources to a single instance. Scaling horizontally means adding more instances to a system.
Use the scaling comparison to see why adding servers can improve redundancy and potential durability, while still requiring coordination and failure handling. section 4.2
