6 Brokers

So far in our discussions, we have been dealing with Kafka from the view of an application developer interacting from external applications and processes. However, Kafka is a distributed system that needs attention to run in its own right. Let’s look at the parts that make the Kafka brokers work!

6.1 Introducing the Broker

Some queues have the concept of a 'smart' broker and a 'dumb' client or the reverse: a 'dumb' broker and a 'smart' client. The smarter piece being the one that includes the most logic - including business rules ^[33]. These terms might fit other queue and/or topic systems well but I would not spend time trying to make Kafka brokers and clients fit in either category. In my opinion, I think that both parts have a lot of power even though the focus in this chapter switches to the brokers. Being a broker in Kafka means being a part of a cluster of machines.

6.2 Why Kafka needs ZooKeeper

6.3 The Job of a Kafka broker

6.4 Configuration at the Broker Level

6.4.1 Kafka’s Other Logs: Application Logs

6.4.2 Application Logs

6.5 Controllers Duties

6.6 Partition Replica Leaders and their role

6.6.1 Inter-Broker Communications

6.6.2 The Role of Replicas

6.7 In-Sync Replicas (ISRs) Defined

6.8 Unclean Leader Election

6.9 Metrics from Kafka

6.9.1 Cluster Maintenance

6.9.2 Adding a Broker

6.9.3 Upgrading your Cluster

6.9.4 Upgrading your clients

6.9.5 Backups

6.10 A Note on Stateful Systems

6.11 Exercise

6.12 Summary