liveBook · Manning

Kafka-specific metric

As explained in Apache Kafka in Action

A Kafka Streams metric is a measurement describing the performance or operation of a Kafka Streams application.

A Kafka-specific metric measures the performance or operation of a Kafka Streams application. It helps reveal processing failures, throughput, latency, dropped records, and state-store conditions before they become service disruptions. section 15.3.4 section 15.6

Use the metric groups to locate where stream processing is slowing or failing:

  • Task-level metrics expose unprocessed messages, including the total number of dropped records. section 15.3.4
  • Processor-node metrics report records consumed and produced, processed-message counts, and average processing time. section 15.3.4
  • State-store metrics describe store conditions such as the number or size of RocksDB memory-table entries. section 15.3.4
  • Stream-level metrics include failed stream threads, while thread-level metrics include processed-record rate. section 15.3.4
Table 15.5 The most important Kafka Connect and Kafka Streams metrics

Java MBean Name

Description

Implications

kafka.connect:type=
connect-worker-metrics,
connector="{connector}",
name=connector-failed
-task-count

The number of failed connector tasks

This metric indicates serious problems as a stopped task means that no messages are processed by the connector. The problem could be temporary (e.g., database maintenance), meaning that the task just needs to be restarted or could be caused by a processing error.

kafka.streams:type=stream
-metrics,client-id=
([-.\w]+),name=failed
-stream-threads

The number of failed stream threads

This metric indicates serious problems as a failed stream thread means that the processing of messages is stopped.

kafka.streams:type=stream
-thread-metrics,thread
-id=([-.\w]+),name=
process-rate

The rate of processed records

This metric depends strongly on the application/client. Monitoring that metric can help detect anomalies in client behaviors, which are very often caused by malfunctioning applications.

kafka.streams:type=stream
-task-metrics,thread-id=
([-.\w]+),task-id=
([-.\w]+),name=dropped
-records-ratel

The rate of dropped records

This metric indicates critical problems as records couldn’t be correctly processed.

Browse the metric table for Java MBean names, descriptions, and operational implications, including failed stream threads and processing rate. section 15.3.4

Detecting a stopped stream

A rising failed-stream-thread count signals a serious processing problem because a failed stream thread stops message processing. Pair it with processed-record rate and processing duration to distinguish a stopped application from a merely slow one. section 15.3.4 section 15.6

How Apache Kafka in Action covers it

How deep it goes
Descends from metric groups to task, processor-node, stream-thread, and state-store measurements, without defining alert thresholds or instrumentation procedures.
Examples
Uses dropped-record totals, failed-stream-thread counts, processed-record rate, processing duration, and RocksDB memory-table sizes.
What it assumes
Assumes familiarity with Kafka Streams applications, tasks, processor nodes, stream threads, and state stores.
What you can do after
Diagnose/debug with it by locating stopped stream threads, unprocessed records, slow processors, and state-store resource conditions.
How it connects
Links Kafka Streams metrics with Kafka Connect metrics, Kafka broker metrics, client metrics, processor nodes, state stores, and RocksDB.
Cover of Apache Kafka in Action
liveBookmanning.com
Read Apache Kafka in Action on liveBook Get the book at manning.com

Also explained in

Compare how 2 books explain Kafka-specific metric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.