A Kafka controller provides the cluster’s control plane: it maintains metadata, coordinates broker operations, monitors broker health, and helps keep the cluster available. KRaft controllers use the Raft consensus protocol, with one active controller and other controllers in hot standby for failover. section 1.3.2
The controller’s core responsibilities include:
- Store and replicate the metadata log, including topic creation, partition assignments, broker registrations, and other cluster-state changes. section 1.3.2
- Detect broker failures through heartbeat messages; a broker that misses the defined timeout may be considered failed and removed from the cluster. section 1.3.2
- Coordinate cluster activities while brokers continue handling client traffic, message processing, storage, and replication. section 1.3.2
A server can combine broker and controller roles, or a deployment can use a separate controller cluster. The choice affects sizing, fault tolerance, and operational planning; controller quorum configuration becomes especially important when moving from ZooKeeper-based architectures to KRaft. section 1.3.2 section 9.0
Enterprise deployment decision
Before exposing a prototype to other teams, an architecture group must budget for more than brokers. Controllers must store topics, configurations, partition details, and other operational metadata, while quorum-based decisions preserve fault tolerance. The team should therefore plan controller placement, incident behavior, and migration steps alongside its on-premises, cloud, or hybrid deployment model. section 9.0
Figure 1.4 Structure of a Kafka cluster: brokers handle client traffic; KRaft controllers manage metadata and coordination.
Use the cluster diagram to distinguish the broker layer, which handles client traffic, from the KRaft controller layer, which manages metadata and coordination. section 1.3.2
Controllers also influence cluster scalability. KRaft supports clusters with millions of partitions, although increasing partitions raises broker resource demands and can lengthen leadership re-election after broker failure. section 2.4.6
For upgrades, back up controller configuration, validate changes in a production-like environment, and preserve the rolling sequence: KRaft controllers, brokers, then supporting components. section 11.1
