Kafka metadata describes cluster structure and operational state. It includes topics, configurations, partition assignments, broker registrations, and other information required to coordinate the cluster. section 1.3.2 section 9.0
Metadata supports several architectural responsibilities:
- KRaft controllers store and replicate a metadata log containing cluster-state changes such as topic creation and partition assignments. section 1.3.2
- Controllers coordinate broker operations and monitor broker health through heartbeat messages, allowing failed brokers to be detected and removed. section 1.3.2
- Topic metadata documents names, partition counts, replication choices, and partitioning strategies so teams share a consistent architecture. section 2.2.4
Customer 360 topic metadata
An AsyncAPI definition records customerProfile at address CUSTOMER_PROFILES and transactions at address TRANSACTIONS, with nine Kafka partitions for each topic. Storing the YAML or JSON in Git keeps these architectural decisions versioned and shareable. section 2.2.4
Figure 1.4 Structure of a Kafka cluster: brokers handle client traffic; KRaft controllers manage metadata and coordination.
Use the cluster diagram to distinguish brokers handling client traffic from KRaft controllers managing metadata and coordination. section 1.3.2
