A Kafka cluster is a group of brokers—the servers that receive, store, and serve messages. In a bank-transfer system, an application produces transfer events to a topic; the cluster distributes that topic’s partitions across brokers so consumers can process the events in parallel. section 1.3 section 4.5
Multiple brokers make replication possible. Each partition has a leader responsible for coordinating it and follower replicas that improve resilience. Two brokers can replicate data, but maintenance of one leaves the cluster exposed if the other fails. Start with at least three brokers; add more when load calls for greater performance. section 1.3 section 4.3.2
A distributed cluster also needs agreement about its state: which brokers are healthy, where partitions belong, and what configuration is current. A controller directs brokers using cluster metadata. Kafka uses KRaft for coordination; older deployments use ZooKeeper. section 7.0 section 1.3
Connect without depending on one broker
To list topics in a three-broker local cluster, run `kafka-topics.sh --list --bootstrap-server localhost:9092,localhost:9093,localhost:9094`. A client can initially contact any listed broker to request metadata about the cluster. Listing only one broker works for a test, but new clients cannot start through that address if that broker is down. At larger scale, a load balancer or DNS can provide the initial contact; individual brokers must still have reachable addresses. section 7.4
Figure 7.3 Before our clients can produce or consume messages, they need to know which broker is responsible for what. This is achieved through a metadata request.
Follow the metadata request that tells a client which broker is responsible before it produces or consumes messages. section 7.4
Operating the cluster requires reliable servers, adequate broker-to-broker networking, suitable topic and partition design, and monitoring. A managed service removes some infrastructure work, not the need to understand the streaming architecture. Protect access and capacity too: TLS listeners support a migration from unsecured connections, while quotas can curb excessive client load—but limits set too aggressively can disrupt service. section 1.4 section 13.4
Prefer one centralized cluster unless requirements justify more. Separate clusters can isolate high-throughput workloads, sensitive data, or regional processing, but each additional cluster increases management and maintenance effort. section 18.4.4
