Log end offset

As explained in Apache Kafka in Action

A log end offset is the position immediately after the last message received in a Kafka partition log or replica log.

A log end offset (LEO) marks the end of the messages received on a Kafka partition log or replica log. Each partition replica maintains its own LEO, allowing Kafka to track how far replication has progressed. section 8.4.1

LEO is central to in-sync replica management. A follower must retrieve messages through the leader’s LEO within the permitted lag time; otherwise, the leader removes it from the partition’s ISR. A replica that stops responding to heartbeats can also be removed immediately. section 8.4.1

LEO participates in replication as follows:

• When a leader or follower receives a message, it advances its corresponding LEO. section 8.4.2
• A follower includes its current LEO in fetch requests, telling the leader which messages it has received and which remain missing. section 8.4.2
• The leader compares replica positions to determine whether followers have caught up and should remain in the ISR. section 8.4.1 section 8.4.2

LEO differs from the High Watermark (HWM). LEO shows how far an individual log has received messages; HWM marks the position replicated by all current ISRs. Consumers can read only through the HWM, so LEO may be ahead while newly received messages remain unavailable for consumption. section 8.4.2

Leader and follower fetch cycle

A partition leader receives a message and advances its LEO. During the next follower fetch, the leader sees that the follower’s LEO is behind and sends the missing message. After a later fetch confirms that the follower has caught up, the replicated position can contribute to the HWM. section 8.4.2

Figure 8.6 This shows the follower’s LCO before and after the fetch request (Current Position). The HWM indicates the latest offset committed by all ISRs. Because the follower is behind, its LCO is lower than the HWM. The LEO is a bit ahead of the HWM, as new messages are arriving.

Compare a follower’s position with the HWM and the leader’s slightly ahead LEO to see how replication lag and newly arriving messages coexist. section 8.4.2

How Apache Kafka in Action covers it

How deep it goes

Explains per-replica positions, follower fetch exchange, ISR lag detection, and the distinction from the High Watermark.

Examples

Uses a leader-follower fetch cycle and a diagram contrasting follower position, LEO, and High Watermark.

What it assumes

Assumes partition logs, leaders, followers, replication, and offsets, while defining LEO and relating it to the High Watermark.

What you can do after

Diagnose/debug replica lag by comparing LEO values with ISR membership and the High Watermark.

How it connects

Connects log end offset with Kafka partitions, replication, leaders, followers, in-sync replicas, and the High Watermark.

liveBookmanning.com

Read Apache Kafka in Action on liveBook Get the book at manning.com

Related concepts

• offsetkind of · An offset is a position or displacement measured relative to a reference point, such as within an ordered sequence, memory region, or layout.
• fetch requestuses · A fetch request is a Kafka protocol request that asks a broker to return records from a topic partition beginning at a specified offset.
• replicameasures · A replica is a copy of data or state, or an equivalent running instance of a component, maintained alongside others in a replicated system.
• partitionpart of · A partition is a distinct subdivision of a larger data, storage, or work unit that can be stored, managed, or processed separately.

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.