A Kafka broker is a server process that forms Kafka’s storage layer. Producers send records to it, consumers request records from it, and the broker stores and returns those records without needing producer–consumer awareness. section 1.4 section 1.6
A broker’s main responsibilities include:
- Storing records durably as byte-formatted key-value data with metadata, while remaining agnostic about their contents. section 1.4.1
- Managing retention and replication, including copying data across brokers so a machine failure does not necessarily lose records. section 2.1
- Handling produce, fetch and metadata requests, processing incoming requests in first-in, first-out order through request handlers. section 2.10
- Persisting records in partitioned topics, which distribute storage and workload across brokers and support throughput. section 2.11
Figure 1.4 You deploy brokers in a cluster, and brokers replicate data for durable storage.
Use the cluster diagram to see brokers replicating records for durable storage. section 1.4.1
A produce-and-fetch exchange
A producer sends a produce request containing records; the broker appends them to a topic. Later, a consumer sends a fetch request with an offset, and the broker retrieves the corresponding records. Consuming does not shorten broker retention. section 2.10 section 4.1
