9 Administration: cluster tools, logging, and monitoring

We have spent time focused on brokers in depth in chapter 6 as well as other client concerns throughout the earlier chapters. But even using some of the best practices discussed, your environment might not always fit the ideal situation. The best way to keep your cluster moving along is to get to know your specific data as well as monitoring what is actually going on in your clusters. Even though running Kafka is not exactly like writing your own Java applications per se, it still requires monitoring log files and looking at what is happening at runtime with your workloads.

9.1  Administration clients

So far, we have taken almost all of the actions that have changed our cluster with the command line tools that are bundled with Kafka. Overall, you will likely need to be comfortable with a shell environment to set up and install Kafka. But there are some useful options that can help if you want to branch out from these provided scripts.

9.1.1  Administration in code with AdminClient

9.1.2  Kafkacat

9.1.3  Confluent REST Proxy API

9.2  Running Kafka as a systemd Service

9.3  Logging

9.3.1  Kafka application logs

9.3.2  Zookeeper logs

9.4  Firewall

9.4.1 Advertised listeners

9.5  Metrics

9.5.1  JMX Console

9.5.2  Important broker metrics

9.5.3  Important producer metrics

9.5.4  Important consumer metrics

9.5.5  Burrow for Consumer Lag Logic

9.6  Tracing messages across a cluster

9.6.1  Producer interceptor

9.6.2  Consumer Interceptor

9.6.3  Overriding Clients