liveBook · Manning

Consumer.committableSource

As explained in Akka in Action, Second Edition

Consumer.committableSource is an Alpakka Kafka source that emits records together with committable offset information for at-least-once processing.

Consumer.committableSource reads Kafka records together with committable offsets, preventing a crash during processing from silently losing the current message. It supports at-least-once processing: a message may be processed again after recovery, but it is not treated as safely complete before its offset is committed. section 14.1.3 section 14.5

Use the source when processing completion must control Kafka offset commitment:

  • Pair the retained offset with a downstream message and Producer.committableSink when reading from Kafka and writing to Kafka. section 14.2.2

Forward records while preserving their offsets

A Kafka-to-Kafka stream can map each incoming message to ProducerMessage.single(new ProducerRecord("test4", msg.record.key, msg.record.value), msg.committableOffset), then materialize it with Producer.committableSink. The offset travels as the envelope's passThrough and is committed by the sink after producing the output record. section 14.2.2

val consumerSettings: ConsumerSettings[String, String] =
      ConsumerSettings(
        config,
        new StringDeserializer(),
        new StringDeserializer())
         .withBootstrapServers("127.0.0.1:9092")
         .withGroupId("group01")
         .withProperty(                                             #1
               ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true")    #1

Compare this explicit commitment model with automatic offset commits: auto-commit can mark a message complete before its processing finishes, allowing a crash to lose it. section 14.1.3

How Akka in Action, Second Edition covers it

How deep it goes
Explains offset-loss failure scenarios and demonstrates offset propagation into a committable sink, without detailing consumer internals.
Examples
Uses a Scala stream that reads topic test3 and writes records to test4 while carrying each source offset forward.
What it assumes
Assumes familiarity with Kafka topics and consumer settings while demonstrating offset commitment through a concrete producer pipeline.
What you can do after
Apply it via Alpakka Kafka to build Kafka pipelines that preserve offsets and provide at-least-once delivery.
How it connects
Connects Kafka consumption to ConsumerSettings, CommittableMessage, committableOffset, ProducerMessage, and Producer.committableSink.
Cover of Akka in Action, Second Edition
liveBookmanning.com
Read Akka in Action, Second Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Akka in Action, Second Edition. The section links point to the parts of the book it draws on.