chapter two

2 Kafka brokers

 

This chapter covers

  • Explaining how the Kafka broker is the storage layer in the Kafka event streaming platform
  • Describing how Kafka brokers handle requests from clients for writing and reading records
  • Understanding topics and partitions
  • Using JMX metrics to check for a healthy broker

In chapter 1, I provided an overall view of the Kafka event streaming platform and the different components that make up the platform. This chapter will focus on the system’s heart, the Kafka broker. The Kafka broker is the server in the Kafka architecture and serves as the storage layer.

In describing the broker behavior in this chapter, we’ll get into some lower-level details. It’s essential to cover them to give you an understanding of how the broker operates. Additionally, some of the things we’ll cover, such as topics and partitions, are essential concepts you’ll need to understand when we get into the client chapter. But as a developer, you won’t have to handle these topics daily.

2.1 Introducing Kafka brokers

As the storage layer, the broker manages data, including retention and replication. Retention is how long the brokers store records. Replication is how brokers make copies of the data for durable storage, meaning you won’t lose data if you lose a machine.

But the broker also handles requests from clients. Figure 2.1 shows the client applications and the brokers.

Figure 2.1 Clients communicating with brokers
figure

2.2 Produce requests

2.3 Fetch requests

2.4 Topics and partitions

2.4.1 Offsets

2.4.2 Determining the correct number of partitions

2.5 Sending your first messages

2.5.1 Creating a topic

2.5.2 Producing records on the command line

2.5.3 Consuming records from the command line

2.5.4 Partitions in action

2.6 Segments

2.6.1 Data retention

2.6.2 Compacted topics

2.6.3 Topic partition directory contents

2.7 Tiered storage

2.8 Cluster metadata