8 Monitoring
This chapter covers
- Observing the health of Kafka Connect connectors
- The most important metrics of Kafka Connect
- Monitoring Kafka Streams applications
- Detecting consumer applications that are lagging behind
The last chapter covered preparing Kafka Streams applications for production and running them on Kubernetes, a container-based orchestration system. It also discussed how to improve the robustness of Kafka Connect deployments. While packaging and deploying streaming applications is an essential part of running them in production, the journey doesn’t end here. Once the application is up and running, we need to continuously monitor it to make sure it’s operating as expected.
Streaming data pipelines consist of many different components and build on top of various technologies. We use Apache Kafka to store event data, implement connectors with Kafka Connect, and build stateful stream processing with Kafka Streams. Whether it’s a Kafka Streams app, a Kafka Connect connector, or any other Kafka workload, these applications continuously process data and move them between systems around the clock. Manually overseeing streaming data pipelines is not an option so we should build automated means to monitor their behavior and health. Fortunately, the different technologies expose metrics that can be used to determine the current operational state.