High-water mark

As explained in Apache Kafka in Action

A high water mark is the highest offset in a Kafka partition whose record is available for consumption because the required replicas have received it.

The high water mark (HWM) is the boundary between replicated, committed records and records still awaiting replication in a Kafka partition. Consumers can read only through this boundary, so a newly produced record may briefly exist beyond the HWM before becoming available. section 8.4.2 section 8.5

The leader calculates the HWM from replica progress. Each leader and follower advances its Log End Offset (LEO); the HWM corresponds to the smallest committed LEO among the in-sync replicas (ISRs). The leader shares this position with followers during fetches. section 8.4.2

A record moves toward consumer visibility through this flow:

• The partition leader receives the record and advances its LEO. section 8.4.2
• Followers fetch missing records and report their current LEO to the leader. section 8.4.2
• After the required ISRs have received the record, the HWM advances and consumers can read it. section 8.4.2

When a replica falls behind

Suppose an ISR can no longer replicate quickly enough. While it remains in the ISR set, it holds the HWM back and slows or blocks near-real-time consumption. Once it exceeds the configured maximum lag and leaves the ISR set, the HWM may jump forward. If fewer replicas than min.insync.replicas remain in sync, the HWM stops advancing, so consumers cannot read new records; producers using acks=all also cannot publish. section 8.4.3

Figure 8.5 The High Watermark is the offset that all ISRs have replicated and committed to the leader. The lowest follower isn’t in sync and isn’t taken into account when calculating the HWM.

Visualizes the HWM as the offset replicated by all relevant ISRs while excluding a follower that is no longer in sync. section 8.4.2

How Apache Kafka in Action covers it

How deep it goes

Explains HWM calculation through LEO and ISR fetches, then traces lag effects without detailing broker persistence internals.

Examples

Illustrates leader and follower offsets, a lagging ISR, HWM advancement, and blocked consumption when the minimum ISR is unavailable.

What it assumes

Assumes familiarity with Kafka partitions, leaders, followers, replication, offsets, and in-sync replicas.

What you can do after

Diagnose/debug with it by relating consumer stalls and delayed visibility to ISR lag and minimum-replica settings.

How it connects

Connects high water mark with replication, in-sync replicas, Log End Offset, consumer availability, min.insync.replicas, and acks=all.

liveBookmanning.com

Read Apache Kafka in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionA high-water mark is the greatest partition offset replicated to all in-sync replicas and made accessible to consumers.

Compare how 2 books explain high-water mark →

Related concepts

• offsetmeasures · An offset is a position or displacement measured relative to a reference point, such as the start of a sequence, an address base, or a default location.
• in-sync replicaused by · An in-sync replica is a Kafka partition replica sufficiently caught up with its leader to remain eligible for failover.
• Reactconfigured by · React is an open-source JavaScript library for building user interfaces from declarative, reusable components.
• min.insync.replicasconfigured by · min.insync.replicas is a Kafka setting specifying the minimum number of in-sync replicas that must acknowledge a write for it to succeed.
• clientconfigures · A client is a person, organization, device, or software component that receives or uses a product, service, resource, or functionality provided by another entity.

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.