The metadata log, named __cluster_metadata, is Kafka’s internal topic for storing the cluster metadata required to operate a KRaft cluster. It replaces the metadata previously held in ZooKeeper and gives brokers a consistent view of cluster state. section 7.1 section 7.5
The log supports coordination and failure handling by:
- Replicating metadata across all brokers, regardless of whether they run controller, broker, or combined roles. section 7.1
- Supporting controllers as they manage partition assignments, leader elections, and failover after broker or controller failures. section 7.1 section 7.5
- Reducing operational dependence on a separate ZooKeeper system by integrating coordination into Kafka through KRaft. section 7.5
Cluster metadata flow
In a KRaft deployment, controllers coordinate cluster state while __cluster_metadata is replicated across the brokers. If the active controller fails, a standby controller takes over and continues managing assignments and leadership using the shared metadata. section 7.1
Figure 7.1 With Kafka 3.3, ZooKeeper was replaced by a coordination cluster based on the Raft consensus protocol.
Use the architecture figure to relate the metadata log to the Raft-based coordination cluster that replaced ZooKeeper. section 7.1
