Consensus lets distributed Kafka nodes agree on shared cluster metadata despite failures. It determines who controls metadata, assigns partitions, elects leaders, and maintains consistent state as brokers join or leave. section 7.0 section 7.5
Keeping a Kafka cluster available
Use an odd-numbered coordination cluster so decisions can reach a majority. Three nodes continue after one failure; five nodes tolerate two failures while maintaining quorum. Smaller Kafka deployments commonly use three KRaft brokers or ZooKeeper nodes, while larger deployments may use five dedicated coordination instances. section 4.3.1
Kafka has used ZooKeeper and now uses KRaft, which applies the Raft protocol inside Kafka brokers. KRaft stores consistent metadata in the __cluster_metadata topic, removes the separate ZooKeeper dependency, and simplifies coordination. Dedicated coordination clusters remain suitable for larger production environments, while smaller clusters can combine broker and coordination duties. section 7.5 section 4.3.1
Figure 4.11 Kafka uses either a KRaft-based or a ZooKeeper-based coordination cluster. Both should consist of an odd number of nodes (usually three or five) and use a protocol to find a consensus.
Compare KRaft- and ZooKeeper-based coordination clusters, their odd-node deployment pattern, and their protocols for reaching consensus. section 4.3.1
