High-water mark

As explained in Kafka Streams in Action, Second Edition

A high-water mark is the greatest partition offset replicated to all in-sync replicas and made accessible to consumers.

The high-water mark is the greatest partition offset stored by every in-sync replica. Kafka exposes consumers only to records at or below this boundary, preventing recently read records from disappearing after a replica failure. section 2.9.1

Pending replication at offsets 8–10

If offsets 8–10 exist on the leader but have not reached every in-sync replica, offset 7 is the high-water mark. Consumers can read through 7; if the leader fails, the producer receives no acknowledgment for the unavailable records and retries them. section 2.9.1

Use the high-water mark alongside these related boundaries:

• In a nontransactional environment, the last stable offset equals the high-water mark, so consumers can poll through it. section 4.4.3
• With transactions, read_committed consumers stop at the last stable offset, which may be below the high-water mark while a transaction remains open. section 4.4.3

Figure 4.24 High water mark vs. last stable offset in a transactional environment

Compare the high-water mark with the last stable offset in a transactional environment. section 4.4.3

How Kafka Streams in Action, Second Edition covers it

How deep it goes

Defines the replication boundary, works through offsets 8–10, and contrasts it with the transactional last stable offset.

Examples

Uses offsets 8–10 awaiting replication and contrasts nontransactional polling with read_committed polling.

What it assumes

Assumes partition replication, leaders, followers, in-sync replicas, offsets, and basic producer-consumer behavior.

What you can do after

Configure consumers with read_committed while distinguishing replication durability from transactional visibility.

How it connects

Connects replicas, in-sync replicas, consumers, producer acknowledgments, transactions, and the last stable offset.

liveBookmanning.com

Read Kafka Streams in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Apache Kafka in ActionA high water mark is the highest offset in a Kafka partition whose record is available for consumption because the required replicas have received it.

Compare how 2 books explain high-water mark →

Related concepts

• offsetmeasures · An offset is a position or displacement measured relative to a reference point, such as the start of a sequence, an address base, or a default location.
• in-sync replicaused by · An in-sync replica is a Kafka partition replica sufficiently caught up with its leader to remain eligible for failover.
• Reactconfigured by · React is an open-source JavaScript library for building user interfaces from declarative, reusable components.
• min.insync.replicasconfigured by · min.insync.replicas is a Kafka setting specifying the minimum number of in-sync replicas that must acknowledge a write for it to succeed.
• clientconfigures · A client is a person, organization, device, or software component that receives or uses a product, service, resource, or functionality provided by another entity.

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.