A three-broker Kafka cluster may work well until a broker fails. Reliability asks whether messages remain stored, producers can keep writing, and consumers can keep reading under those conditions. It also asks whether writes are persisted as intended. These are distinct concerns: durability, availability, and consistency. section 5.0
The concerns call for different safeguards:
- Replication across brokers supports durability and availability when a broker fails. section 5.0
- Acknowledgments, idempotent writes, and transactions help producers address whether messages arrive and multiple writes retain their intended integrity. section 5.0
- Capacity and placement matter too: separate brokers across hosts or racks, provide suitable disks and network capacity, and avoid relying on a single machine. section 14.4 section 4.2
An acknowledgment is not necessarily a disk write
Kafka can acknowledge a write after placing data in the operating system’s page cache, before it reaches disk. A power failure could lose that copy. Replicating data across brokers provides protection, provided they do not all fail and lose it at once. An ACK therefore needs to be understood alongside replication, not mistaken for proof of an immediate disk commit. section 6.3
Figure 4.3 Scaling vertically means adding more resources to a single instance. Scaling horizontally means adding more instances to a system.
Compare adding resources to one server with adding servers: a stronger single server may improve performance without meaningfully reducing its chance of failure. section 4.2
Keep checking the deployed system. Under-replicated partitions and partitions below the minimum in-sync replica count should ideally be zero. Watch client errors and delays as well as broker capacity; leave room for rebalancing after failures. Set quotas carefully, because limits intended to protect the cluster can themselves disrupt service. section 15.6 section 13.4
