Kafka broker

As explained in Kafka for Architects

A Kafka broker is a Kafka server that stores, receives, and serves records for producers and consumers without interpreting their application-level content.

A Kafka broker is a Kafka server that stores, receives, and serves records for producers and consumers. It treats each record as opaque bytes, forwarding application-level interpretation to clients. section 1.5.1 section 4.2

A production deployment normally uses a cluster of brokers. Brokers handle client traffic, storage, and replication, while KRaft controllers maintain metadata, coordinate cluster operations, and detect failed brokers. section 1.3.2

Consumer polling

A consumer requests partition assignments, polls a broker repeatedly, receives available records in batches, deserializes and processes them, then commits its offset. The offset records where reading resumes; it does not confirm successful processing of each record. section 4.2

Broker opacity keeps Kafka fast and flexible but prevents broker-side content-based routing and message validation. Clients must deserialize and validate records, or an intermediate service such as Kafka Streams or Apache Flink must route them. section 5.2.3 section 1.5.1

Figure 1.4 Structure of a Kafka cluster: brokers handle client traffic; KRaft controllers manage metadata and coordination.

Use the cluster view to distinguish broker responsibilities from KRaft controller responsibilities. section 1.3.2

How Kafka for Architects covers it

How deep it goes

Explains cluster roles, opaque records, consumer polling, offsets, and broker limitations, but defers detailed schema and compatibility design.

Examples

Uses a broker cluster diagram, Schema Registry message flow, and consumer polling, processing, batching, and offset-commit workflow.

What it assumes

Assumes familiarity with producers, consumers, messages, and event-driven communication while explaining broker and controller roles.

What you can do after

Operate/configure broker-based flows while placing deserialization, validation, routing, and processing in clients or intermediary services.

How it connects

Connects brokers with producers, consumers, partitions, offsets, KRaft controllers, Schema Registry, Kafka Streams, and Apache Flink.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Apache Kafka in ActionA Kafka broker is a server process that stores and serves Kafka records and participates in cluster operations.
• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionA Kafka broker is a server process that stores Kafka records and handles requests from Kafka clients and consumers.

Compare how 3 books explain Kafka broker →

Related concepts

• transaction coordinatorincludes · A transaction coordinator is a component that coordinates distributed transaction participants and directs them to commit or roll back as a unit.
• binary searchuses · Binary search is an algorithm that locates a target or boundary in an ordered collection by repeatedly comparing with a midpoint and halving the remaining search range.
• serverkind of · A server is a computer system or software process that receives requests from clients or other systems and provides resources, data, or services in response.
• clientused by · A client is a person, organization, device, or software component that receives or uses a product, service, resource, or functionality provided by another entity.
• primary nodeincludes · A primary node is a designated authoritative or coordinating member of a distributed system, typically responsible for accepting operations or coordinating replicated state.
• Kafka clusterpart of · A Kafka cluster is a group of Kafka brokers that collectively store, distribute, and serve event streams.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.