6 Brokers

This chapter covers:

Exploring the role of brokers, and their duties.
Evaluating options for certain broker configuration values.
Explaining replicas and how they work to stay up-to-date.
Assessing what JMX is and what it can provide.

So far in our discussions, we have been dealing with Kafka from the view of an application developer interacting from external applications and processes. However, Kafka is a distributed system that needs attention to run in its own right. Let’s look at the parts that make the Kafka brokers work!

6.1 Introducing the Broker

While we have focused on the client-side of Kafka so far, our focus will now shift to another powerful component of the ecosystem: brokers. Being a broker in Kafka means being a part of a cluster of machines - brokers work together with other brokers to form the core of the system.

6.2 Role of ZooKeeper

6.3 The Job of a Kafka broker

6.4 Settings at the Broker Level

6.4.1 Kafka’s Other Logs: Application Logs

6.4.2 Server Log

6.5 Controllers Duties

6.6 Partition Replica Leaders and their role

6.7 In-Sync Replicas (ISRs) Defined

6.8 Unclean Leader Election

6.9 Metrics from Kafka

6.9.1 Cluster Maintenance

6.9.2 Adding a Broker

6.9.3 Upgrading your Cluster

6.9.4 Upgrading your clients

6.9.5 Backups

6.10 A Note on Stateful Systems

6.11 Exercise

6.12 Summary