Consumer lag

As explained in Kafka for Architects

Lag is the amount by which processing or consumption falls behind the latest available data.

Consumer lag measures how far a consumer group falls behind the latest records available in a Kafka partition. It exposes the group’s unprocessed backlog and helps determine whether consumers are keeping pace with changing workload patterns. section 4.3.1

Measure and interpret consumer lag in two main ways:

• Use Kafka’s JMX consumer-lag metric. section 4.3.1
• Approximate lag by subtracting the consumer group’s latest committed offset from the latest produced offset for a partition. Missing offsets, especially after compaction, can make this calculation imprecise. section 4.3.1
• Use the trend in lag to decide whether to scale consumer instances and to investigate throughput bottlenecks or processing errors. section 4.3.1 section 8.5.6

Customer 360 processing

An ODS consumer aggregates customer profile and transaction topics by customer ID. Monitoring consumer lag shows whether the service keeps up with incoming data; rising lag signals a processing backlog and may justify additional consumer instances, subject to partition limits. section 4.0 section 4.3.1

Figure 4.20 Calculating consumer lag

A partition-offset diagram clarifies how committed and produced positions determine consumer lag. section 4.3.1

How Kafka for Architects covers it

How deep it goes

Defines lag, gives JMX and offset-based calculations, and notes missing-offset inaccuracies without detailing metric implementation.

Examples

Uses a customer aggregation consumer and a partition-offset diagram to make backlog measurement concrete.

What it assumes

Assumes familiarity with Kafka partitions, consumer groups, produced records, committed offsets, and basic monitoring.

What you can do after

Diagnose processing backlogs and evaluate when consumer instances or monitoring investigation may be necessary.

How it connects

Connects consumer lag with consumer groups, partitions, offsets, Kafka Streams, throughput, monitoring, and distributed tracing.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Apache Kafka in ActionA consumer lag metric reports replication lag separately for each follower, topic, and partition.

Compare how 2 books explain consumer lag →

Related concepts

• CQRSused by · CQRS is an architectural pattern that separates operations and models for changing application state from those for querying or reading state.
• monitoringmeasured by · Monitoring is the ongoing observation and measurement of a system’s behavior, state, or operational signals to assess its condition and detect changes or problems.
• consumer groupmeasures · A consumer group is a logical set of message consumers that cooperatively processes a shared stream by distributing its partitions among group members.
• offsetuses · An offset is a position or displacement measured relative to a reference point, such as within an ordered sequence, memory region, or layout.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.