Metadata log
A metadata log is an ordered, replicated append-only log recording changes to Kafka cluster metadata.
Kafka for Architects
A Raft log is an ordered replicated log used by Kafka controllers in KRaft mode to store changes to cluster metadata.
- How it is used
- Records Kafka cluster-state changes so KRaft controllers can manage metadata and coordinate with brokers.
- How deep it goes
- Explains controller storage, replication, majority commit, and snapshots at an architectural level rather than detailing log-file formats.
- Examples
- Traces a topic-creation request from an admin client through a broker to the leader controller and replicated log.
- What you can do after
- Evaluate/understand how controller-majority loss affects metadata changes and high-availability topology.
Apache Kafka in Action
__cluster_metadata is Kafka's internal topic for storing cluster metadata across brokers in KRaft mode.
- How it is used
- Coordinates Kafka cluster state in KRaft deployments by providing shared metadata for controllers and brokers.
- How deep it goes
- Explains its purpose, replication scope, and operational role, but does not detail record formats or log internals.
- Examples
- Uses the __cluster_metadata topic, replicated broker state, and active-to-standby controller takeover as concrete examples.
- What you can do after
- Evaluate/understand how KRaft stores and distributes Kafka cluster metadata during coordination and failover.
Related concepts
- follower controllerused by · A follower controller is a KRaft controller that replicates metadata from the active controller and participates in controller quorum decisions.
- KRaftused by · KRaft is Apache Kafka’s Raft-based metadata-management mode, using an internal controller quorum instead of ZooKeeper.
- controllerused by · A Kafka controller is a server role that manages cluster metadata and coordinates partition, replica, and leader state.
- metadatahas part · Metadata is information that describes, identifies, or provides context about other data, resources, or artifacts.
- logkind of · A log is a record of events, activity, or diagnostic information produced during the operation of a system or process, often ordered by time.
- controller clusterpart of · A controller cluster is a group of dedicated Kafka controller nodes that collectively manage cluster metadata and coordinate broker operations using the KRaft consensus protocol.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.