A coordination cluster is the orchestration layer of a Kafka deployment. It monitors brokers, tracks membership and metadata, manages topic access and partition assignments, and helps keep the cluster available when brokers fail. section 4.3.1
Its operational responsibilities include:
- Monitoring broker reachability and maintaining the active-broker list, including broker additions and removals. section 4.3.1
- Storing topic metadata, access information, partition assignments, and the leader broker for each partition. section 4.3.1
- Designating a controller that manages partition and replica state, reassigning partitions and selecting leaders after broker failures. section 4.3.1
- Providing metadata that clients use to discover which brokers lead partitions; clients obtain it through a broker metadata request. section 7.4
Bootstrapping a client
A topic-listing command can start with one broker using --bootstrap-server localhost:9092, but production clients should specify multiple brokers, such as localhost:9092,localhost:9093,localhost:9094. A load balancer or DNS can simplify discovery as the cluster grows. section 7.4
Figure 4.11 Kafka uses either a KRaft-based or a ZooKeeper-based coordination cluster. Both should consist of an odd number of nodes (usually three or five) and use a protocol to find a consensus.
Use the architecture figure to compare KRaft-based and ZooKeeper-based coordination clusters and their consensus-oriented node layout. section 4.3.1
Coordination clusters normally use an odd number of nodes, commonly three or five. Three nodes allow one failure without losing availability; five allow two failures. Coordination nodes need modest compute resources, but reliable, low-latency networking is important. section 14.3.2
