liveBook · Manning

Idempotent consumer pattern

As explained in Microservices Patterns, Second Edition

A messaging pattern in which a consumer detects and discards duplicate messages so repeated delivery has the effect of processing a message once.

Message brokers commonly provide at-least-once delivery, so a consumer may receive a message again after a failure. For example, a client can process a message and update its database, then crash before acknowledging it. section 8.3.7

An idempotent consumer detects and discards duplicates so repeated delivery has no additional effect. One approach records processed message IDs in a database table; inserting an already-recorded ID fails. section 8.3.7

Choose a handling strategy based on whether the message’s application logic is naturally idempotent:

  • Make the handler idempotent when repeated calls with the same input have no additional effect. section 8.3.7
  • Track processed messages and discard duplicates when the handler is not naturally idempotent. Preserve redelivery ordering where needed: reordering an order-creation event and an order-cancellation event can leave the order in the wrong state. section 8.3.7
Figure 8.16 A consumer detects and discards duplicate messages by recording the IDs of processed messages in a database table. If a message has been processed before, the INSERT into the PROCESSED_MESSAGES table will fail.
IdempotentConsumer

The diagram shows duplicate detection through a processed-message table: a repeated message ID makes the insert fail. section 8.3.7

Eventuate’s MessageConsumer implements the pattern, while its MessageProducer uses the transaction outbox for sending. These are distinct responsibilities in the messaging flow. section 8.3.10

How Microservices Patterns, Second Edition covers it

How deep it goes
Explains redelivery causes, contrasts two handling strategies, and illustrates recording processed IDs in a database.
Examples
Uses redelivery of Order Created and Order Cancelled events, plus a processed-message table and Eventuate’s MessageConsumer.
What it assumes
Assumes familiarity with message brokers, consumer acknowledgments, and service databases; explains duplicate delivery through a failure scenario.
What you can do after
Apply it by making handlers idempotent or tracking processed-message IDs and discarding duplicates.
How it connects
Connects message brokers and at-least-once delivery with idempotent message handlers, event ordering, and the transaction outbox.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Microservices Patterns, Second Edition. The section links point to the parts of the book it draws on.