liveBook · Manning

Eager buffering

As explained in Kafka Streams in Action, Second Edition

Eager buffering is a buffering policy that permits early result emission before a window is final, potentially producing duplicate results.

Eager buffering bounds suppression storage by record count or byte size, allowing Kafka Streams to emit intermediate windowed results before the window closes. section 9.3

Use eager buffering when bounded memory matters more than guaranteeing one final result:

  • Configure a time limit for waiting on a new record; a new record replaces the existing buffered result without resetting the timer. section 9.3.2
  • Configure a maximum record count or byte size; when the buffer fills, suppression forwards a result immediately. section 9.3 section 9.3.2
  • Expect possible intermediate or duplicate results rather than the strict strategy’s single final result. section 9.3

Kafka Streams configuration

Apply eager buffering with a one-minute time limit and a 1,000-record maximum: .suppress(untilTimeLimit(Duration.ofMinutes(1), maxRecords(1000).emitEarlyWhenFull())). Matching the time limit to the window size plus any grace period may produce a final result when the window closes, but a full buffer can still emit earlier. section 9.3.2

Listing 9.26 Using suppression with eager buffering
.suppress(untilTimeLimit(Duration.ofMinutes(1),    #1
                           maxRecords(1000)                #2
                          .emitEarlyWhenFull()))    #3

Inspect the Java suppression configuration for combining a time limit, record bound, and early emission. section 9.3.2

How Kafka Streams in Action, Second Edition covers it

How deep it goes
Explains time and size triggers, replacement behavior, window timing, and duplicate-result tradeoffs through Java configuration.
Examples
Uses a Kafka Streams suppression call with a one-minute limit, a 1,000-record buffer, and emitEarlyWhenFull().
What it assumes
Assumes windowed or table aggregation and suppression terminology while contrasting eager buffering with strict buffering.
What you can do after
Operate/configure Kafka Streams suppression with bounded eager buffering and anticipate early or duplicate aggregation results.
How it connects
Connects suppression with windowed aggregation, time limits, grace periods, buffer limits, and final results.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.