Moving a Kafka prototype into a shared environment requires more than brokers. Topics, configurations, partition details, and other operational metadata need a place to live and a way to stay coordinated as the cluster changes. A KRaft controller cluster provides that metadata infrastructure. section 9.0 section 9.1.1
A KRaft controller cluster is a group of servers that stores cluster metadata and coordinates broker operations. KRaft is Kafka’s implementation of the Raft consensus protocol. Controllers keep metadata in filesystem-backed Raft logs, rather than in an ordinary topic replicated by brokers. Only controllers participate in the metadata Raft quorum; brokers may keep a local, read-only copy. section 9.1.1
The metadata includes:
- Topic names, partition counts, replication factors, and configurations. section 9.1.1
- Runtime partition details, including leaders and the brokers hosting replicas. section 9.1.1
Metadata updates go to the leader controller. It appends each update to its log, and follower controllers fetch it. An update is committed only after a majority acknowledge it. This quorum rule means a newly elected controller leader has all committed entries; it also makes retaining a majority of controllers an important deployment constraint. section 9.1.1
Figure 9.3 In the event of a complete data center outage, losing the majority of controllers causes a loss of quorum if the cluster is distributed across two data centers. However, with three data centers, quorum can still be maintained despite the failure of an entire data center. In this case, N = 1, meaning the system can tolerate only one node failure.
The data-center diagram makes the placement trade-off concrete: losing a majority of controllers loses quorum, while distribution across three data centers can preserve it through one complete data-center outage. section 9.1.1
Clients do not read the metadata log directly. They request metadata from brokers; brokers obtain updates from controllers, forward client-initiated metadata changes to the leader controller, and send heartbeats that help the controller track broker liveness. This separation makes controller capacity and placement part of the Kafka deployment plan, even though clients communicate with brokers. section 9.1.1 section 9.0
