How deeply does Kafka Streams in Action, Second Edition cover producer idempotence?

Explains configuration, producer IDs, sequence numbers, broker validation, and the limitation to a single producer session.

Producer idempotence

As explained in Kafka Streams in Action, Second Edition

enable.idempotence is a Kafka producer configuration that enables idempotent record delivery.

Producer idempotence configures KafkaProducer to deliver records in order and only once within a topic partition, preventing retries from creating duplicates or reordering batches. section 4.4 section 4.4.1

Configure the producer with these related settings:

Set enable.idempotence=true. section 4.4.1
Set acks to all, keep retries greater than 0, and keep max.in.flight.requests.per.connection at no more than 5. section 4.4.1
Use unique producer IDs and monotonically increasing sequence numbers assigned per partition and producer session. section 4.4.1

Idempotent KafkaProducer configuration

A Java producer sets ACKS_CONFIG to "all", ENABLE_IDEMPOTENCE_CONFIG to true, RETRIES_CONFIG to Integer.MAX_VALUE, and MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION to 5.

Listing 4.6 KafkaProducer configured for idempotence

Map<String, Object> producerProps = new HashMap<>();
//Standard configs
producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "somehost:9092");
producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, ...);
producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ...);

//Configs related to idempotence
producerProps.put(ProducerConfig.ACKS_CONFIG, "all");   #1
producerProps.put(
  ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);     #2
producerProps.put(
  ProducerConfig.RETRIES_CONFIG,
                  Integer.MAX_VALUE);   #3
producerProps.put(
ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION,
                     5);          #4

Inspect the complete Java configuration listing and its numbered idempotence settings. section 4.4.1

How Kafka Streams in Action, Second Edition covers it

Cover of Kafka Streams in Action, Second Edition

liveBook manning.com

Read Kafka Streams in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Apache Kafka in Actionenable.idempotence is a Kafka producer setting that enables duplicate-safe retries of records.
liveBook manning.com

Kafka for Architectsenable.idempotence is a Kafka producer setting that enables broker-managed deduplication and ordering protection for producer retries.

Compare how 3 books explain producer idempotence →

Related concepts

Kafka transactionused by · A Kafka transaction is an atomic operation that groups record production and related offset commits so their effects are committed together or aborted.
Kafka producerconfigured by · A Kafka producer is an application or client that publishes records to Kafka topics.
transactional producerimplemented by · A transactional producer is a Kafka producer mode that publishes records to multiple topics as one atomic transaction, committing all records together or none of them.
idempotencyconfigures · Idempotency is the property of an operation whereby repeating it has the same effective result or externally observable effect as performing it once.
ordering guaranteeconfigures · An ordering guarantee is a promise that related operations or records are observed or processed in a specified sequence.
exactly-once deliveryimplements · Exactly-once delivery is a messaging or processing guarantee that each message or input produces its intended effect once, without loss or duplicate effects, despite retries or failures.

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.