A Kafka broker is a server that receives, stores, and serves records. Brokers form the Kafka cluster: producers write to them, consumers fetch from them, and partitions spread a topic’s data across them. For a bank-transfer stream, brokers hold the transfer records rather than sending each event directly from the application to every consumer. section 4.5 section 1.3
A broker’s role depends on the partition. Its leader handles that partition’s reads and writes; follower brokers copy records from the leader. Replicas on multiple running brokers let a follower take over if the leader fails. Replication is set per topic, and its factor cannot exceed the number of available brokers. section 4.2.3
Inspect where replicas live
Create a topic with three partitions and replication factor three using kafka-topics.sh --create --topic products.prices.replication --partitions 3 --replication-factor 3 --bootstrap-server localhost:9092. Then run kafka-topics.sh --describe --topic products.prices.replication --bootstrap-server localhost:9092. The output identifies each partition’s Leader, Replicas, and Isr; the three example partitions have leaders on different brokers. A factor of four fails when only three brokers are registered. section 4.2.3
Clients need not know every leader before connecting. They bootstrap through a broker, request cluster metadata, and use it to find partition leaders. For production bootstrapping, supply multiple broker addresses rather than relying on one that might be down. Each broker still needs its own reachable address. section 7.4
Figure 8.3 When a broker receives a produce request (1), it’s put on the request queue (2) and later handled by the I/O threads (3) and written to the page cache (4). The operating system flushes it to disk later. Then, the message stays in purgatory (5) until all in-sync followers acknowledge the message (6). When the wait (7) is over, the response is put on the response queue (8) and sent back to the producer by one of the network threads (9).
Follow a produce request through a broker’s network and I/O threads, page cache, follower acknowledgments, and response queue. An acknowledgment does not necessarily mean the operating system has flushed the record to disk; replication provides the principal protection described here. section 8.2.1 section 6.3
For a self-managed cluster, start with at least three brokers so maintenance and another failure do not leave a two-broker deployment exposed. Add brokers as load grows. Prioritize dependable storage, memory for the operating system’s page cache, network capacity, and monitoring; tune file-descriptor limits and thread settings for demanding production workloads rather than assuming defaults fit every case. section 4.3.2 section 6.3.1 section 14.3.1
