liveBook · Manning

Consumer lag

As explained in Apache Kafka in Action

A consumer lag metric reports replication lag separately for each follower, topic, and partition.

When a broker replicates a partition, its follower can fall behind the leader. The broker-side consumer lag metric, under fetcher lag, reports that gap separately for each follower, topic, and partition. It helps locate replication delay rather than hiding it in a cluster-wide maximum. section 15.2.1

Keep this metric distinct from a consumer group's lag. Group lag counts messages between the group's current offset and the log end offset; it answers whether an application has read the partition. Replica fetcher lag answers whether a follower has kept up with its leader. section 9.3.2 section 15.2.1

Use the broker metrics together:

  • Check max lag for the largest gap between a leader replica and its followers, regardless of which follower, topic, or partition is involved. section 15.2.1
  • Check consumer lag under fetcher lag to identify the particular follower, topic, and partition that is behind. section 15.2.1
  • Expect brief delays, but investigate lag that grows progressively. Ideally, lag stays minimal and does not exceed the maximum production batch size. section 15.2.1

How Apache Kafka in Action covers it

How deep it goes
Defines the metric's scope and operating expectations, but does not give a collection command or alert threshold.
Examples
Uses leader-and-follower replication as its operating case; offset tables illustrate the distinct consumer-group lag.
What you can do after
Diagnose/debug with it by locating a lagging follower, topic, and partition and watching for progressively increasing delay.
How it connects
Contrasts replica fetcher lag with max lag and consumer-group offset lag.

Also explained in

Compare how 2 books explain consumer lag →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.