9 Management: tools and logging

 

This chapter covers:

  • Exploring helpful scripting administration client options
  • Examining REST APIs, tools, and utilities that can be leveraged for management
  • Mananging the logs of Kafka and ZooKeeper
  • Monitoring the cluster with JMX and related metrics
  • Understanding how advertised listeners work with clients
  • Tracing example using interceptors with headers

We have spent some time discussing brokers in depth in Chapter 6 and client concerns throughout the earlier chapters. We’ve seen some best practices that can be applied in most situations, but there will always be environments where special handling is required. The best way to keep a cluster moving along is to understand the data that is flowing through it, and to monitor that activity at runtime. While operating Kafka may not be as involved as writing and running Java applications per se, it still requires monitoring log files and being aware of what is happening with our workloads.

9.1 Administration clients

So far, we have performed most of our cluster management activities with the command line tools that come with Kafka. And in general, we will need to be comfortable with a shell environment to set up and install Kafka. However, there are some helpful options we can use to branch out from these provided scripts.

9.1.1 Administration in code with AdminClient

 
 
 

9.1.2 Kafkacat

 
 

9.1.3 Confluent REST Proxy API

 
 
 

9.2 Running Kafka as a systemd Service

 
 
 
 

9.3 Logging

 
 
 
 

9.3.1 Kafka application logs

 
 
 

9.3.2 Zookeeper logs

 
 
 

9.4 Firewall

 

9.4.1 Advertised listeners

 
 
 

9.5 Metrics

 
 
 

9.5.1 JMX Console

 

9.5.2 Broker alerts

 

9.5.3 Client alerts

 
 

9.5.4 Burrow

 
 

9.6 Tracing with Interceptors

 

9.6.1 Producer interceptor

 
 
 

9.6.2 Consumer Interceptor

 
 
 

9.6.3 Overriding Clients

 

9.7 General monitoring tools

 
 
sitemap

Unable to load book!

The book could not be loaded.

(try again in a couple of minutes)

manning.com homepage
test yourself with a liveTest