This chapter covers:
- Exploring the roles of brokers in our context
- Important metrics to monitor in our cluster
- Evaluating tradeoffs for certain broker configuration values
So far in our discussions, we have been dealing with Kafka from the view of an application developer interacting from external applications and processes. However, Kafka is a distributed system that needs attention to run in its own right. Let’s start to look at the parts that make the message brokers work!
Some queues have the concept of a 'smart' broker and a 'dumb' client or the reverse: a 'dumb' broker and a 'smart' client. These terms might fit other queue and/or topic systems well but I would not get too involved in trying to make Kafka brokers and clients fit in either category. In my opinion, I think that both parts have a lot of power. Being a broker in Kafka means being a part of a cluster of machines.
If you are familiar with Big Data concepts or worked with Hadoop before, you might near similar terminology such as rack awareness and partitions. For example, Kafka has a rack awareness feature that will make replicas of the same partition exist physically across different racks. This is important since if all of the servers that make up your cluster are on the same rack, and the entire rack is offline, it would be the same as if you had lost your entire cluster.