A Kafka cluster is a group of broker nodes that exchange messages through a shared platform rather than through direct application-to-application connections. Producers push messages to brokers; consumers subscribe to topics and pull messages. Brokers persist messages, serve consumers, distribute workloads, and continue providing service when individual nodes fail. section 1.3.1
The cluster separates client traffic from coordination. Brokers process, store, and replicate messages, while KRaft controllers maintain metadata, monitor broker health, and coordinate cluster operations. One controller is active; others remain ready to take over after a failure. section 1.3.2
A cluster provides several architectural capabilities:
- Replicates messages across brokers so another node can serve data after a broker failure. section 1.4.2
- Uses producer acknowledgments and retries to reduce message loss between a sending application and the cluster. section 1.4.2
- Tracks consumer progress so disconnected consumers can resume processing and replay retained messages when necessary. section 1.4.2
- Supports durable local storage and optional tiered storage, keeping recent data on faster storage while offloading older data. section 1.3.1
Failure and recovery
A service sends a message and waits for the cluster’s acknowledgment. If the acknowledgment does not arrive before the timeout, it retries. If a broker fails, replicated copies remain available on other brokers; if a consumer disconnects, it later resumes undelivered messages. section 1.4.2
Figure 1.4 Structure of a Kafka cluster: brokers handle client traffic; KRaft controllers manage metadata and coordination.
Use the cluster structure diagram to distinguish brokers handling client traffic from KRaft controllers managing metadata and coordination. section 1.3.2
Choosing a cluster is an architectural and operational decision, not merely a deployment step. Capacity, service levels, monitoring, security, disaster recovery, scalability, compliance, and the trade-off between managed and self-managed operation shape the design. section 1.6.1 section 9.2
