How deeply does Kafka for Architects cover durability?

Explains persistence, replication, acknowledgments, offset commits, and failure trade-offs; stops short of a low-level storage walkthrough.

Durability in Kafka for Architects

Durability is the property that acknowledged data survives failures and remains available for later retrieval.

Durability means that acknowledged data survives failures and remains available for later retrieval. For a payment event, losing an acknowledged message can be worse than temporarily stopping new writes. Kafka addresses that risk by persisting messages and replicating them across brokers, rather than relying on delivery alone. section 11.3.2 section 1.7.3

Kafka stores events in a distributed commit log. Brokers persist messages to durable local storage and make them available to consumers; replicas provide protection when a broker fails. Storage is not indefinite by default: a streaming topic removes records when its time- or size-based retention period expires. A compacted topic instead retains at least the latest record for each key, making it useful for state such as configuration data. section 1.3.1 section 2.0

Data safety depends on choices across the write and processing path:

Maintain multiple in-sync replicas and use producer acks=all to wait for their acknowledgments before treating a write as stored. section 11.3.2
If too few in-sync replicas remain, accept that a partition may go offline rather than risk data loss or corruption. section 11.3.2
After a consumer processes data, commit its offset manually rather than advancing it before processing is complete. section 11.3.2

Durability is not the same guarantee as exactly-once processing. In database terminology, the durability part of ACID means committed changes remain permanent after failure. Kafka transactions address a different question in a read-process-write pipeline: whether processing an input and producing its outputs can complete atomically despite retries or failures. A durable input record alone does not prevent duplicated or missing downstream results. section 7.4.2

Table 11.2 Balancing durability and availability

Configuration setting	Improve durability	Improve availability	Setting type
`acks`	Use `acks=all` to ensure data is written to all in-sync replicas	With `acks=1` or `0` the producer can continue sending data even if some replicas are temporarily unavailable	Producer
`enable.idempotence`	Enable to avoid duplicate messages and improve reliability	Enable to avoid duplicate messages and retries in the producer	Producer
`enable.auto.commit`	Disable and manually commit offsets after processing to avoid data loss	Consumer progresses without waiting for manual commit logic to complete	Consumer
`isolation.level`	Set to `read_committed` to read only committed data	Set to `read_uncommitted` to allow faster reads of all data	Consumer
`max.poll.records`	Smaller batches reduce memory pressure and risk of data loss during failure	Larger batches improve consumer throughput and avoid stalling (but may risk larger reprocesses)	Consumer
`max.poll.interval.ms`	Reduce to catch slow consumers faster and avoid stale offsets	Increase to allow slower consumers to stay in group longer	Consumer
`session.timeout.ms`	Reduce to detect dead consumers quickly and maintain ISRs	May be increased to avoid unnecessary rebalances, but for detecting dead consumers it can be decreased	Consumer
`replication.factor`	Use a higher replication factor for better data redundancy	A lower replication factor can speed up recovery but risks data loss	Broker
`min.insync.replicas`	Set higher to ensure more replicas are in sync before committing data	Set lower to allow writes even with fewer in-sync replicas	Broker
`unclean.leader .election.enable`	Disable to avoid data loss with unclean leader election	Enable to allow faster leader election even if some data is lost	Broker
`broker.rack`	Distribute replicas across racks to avoid rack failures causing data loss	Ensure rack awareness for faster recovery if a rack fails	Broker
`log.flush.interval .messages`	Reduce to flush data more frequently to disk, minimizing potential data loss	Increase to flush data less frequently for faster throughput	Broker
`log.flush.interval.ms`	Reduce to flush data to disk more frequently	Increase to flush less often and improve broker throughput	Broker
`num.recovery.threads .per.data.dir`	Increase to reduce the window in which durability is weakened	Increase to speed up broker recovery after a crash	Broker
`num.standby .replicas`	Increase to reduce the risk of data loss during failover	Increase to maintain availability during maintenance or failures	Broker
`replica.lag.time .max.ms`	Reduce to quickly detect lagging replicas and exclude them from ISRs for consistency	Increase to tolerate more lag and avoid unnecessary ISR changes that can affect availability	Broker
`auto.leader .rebalance.enable`	Disable to keep leaders stable, avoiding leader changes that can affect data consistency	Enable to allow automatic rebalancing of leaders, improving availability during broker failures	Broker

Compare producer acknowledgment and consumer commit settings when choosing between stronger data safety and higher availability. section 11.3.2

How Kafka for Architects covers it

Also explained in

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.