Chapter 4: Message delivery and processing

Introduction

The chapter introduces message delivery and processing by distinguishing literal exactly-once processing from exactly-once processing semantics and warning against assuming guarantees are broader than stated.

• Exactly-once processing distinction: Distinguish exactly-once processing, in which each message is processed only once, from exactly-once processing semantics, in which the system behaves as though that happened despite possible retries or duplicates.
  Why it matters: The distinction keeps an observable guarantee from being mistaken for a claim about what happens internally.
• Limits of advertised guarantees: Explain why vague or incomplete claims about exactly-once processing semantics can mislead engineers into designing around guarantees that were never provided.
  Why it matters: Checking what a guarantee actually covers helps prevent designs based on false assumptions.

Key terms: exactly-once processing, exactly-once processing semantics, retries, duplicates, mental model, limitations

4.1 Exchanging messages

This section establishes the failure model and request–response dialogue used to distinguish message delivery from processing and to frame the chapter’s once-only semantics.

• Communication and failure assumptions: Describe the chapter’s partially synchronous model: components have private local state, may crash and recover with some state forgotten, and communicate over a network that may reorder, drop, or duplicate messages. State that Byzantine behavior is excluded.
  Why it matters: These assumptions define which failures a message exchange must account for.
• Request response dialogue: Trace the five-step exchange from the sender submitting a request through network delivery, receiver processing, response submission, and response delivery. Explain that processing may change the receiver’s local state, producing a side effect.
  Why it matters: The sequence pinpoints where delivery, processing, and side effects occur.
• Petri net dialogue model: Explain how the section’s Petri net represents the dialogue: places hold control-state tokens, transitions represent events, and an enabled transition fires by consuming input tokens and producing output tokens.
  Why it matters: The notation makes the dialogue’s possible state changes explicit.
• Delivery and processing distinction: Distinguish message delivery—the network transferring and presenting a message to a component—from message processing—the component generating a response and creating a possible side effect.
  Why it matters: Receiving a message and carrying out its application logic are different events with different consequences.
• Once semantics focus: State that the chapter considers at-most-once, at-least-once, and exactly-once behavior for both delivery and processing, but generally focuses on processing because it contains the application logic and side effects.
  Why it matters: The distinction keeps attention on whether application work occurred, not just how messages moved.

Key terms: partially synchronous system model, volatile, durable, Byzantine failures, local state, sender, receiver, request, response, side effect

4.2 The uncertainty principle of message delivery and processing

The section connects a component’s limited local view to the sender’s shift from certainty to uncertainty during message delivery and processing.

• Local view and sender uncertainty: Explain that a component can observe only its own state and its channel to the network, not the whole system, and that this limited view causes a sender to move from certainty to uncertainty, ideally returning to certainty.
  Why it matters: The sender’s limited view explains why it cannot always be certain what happened to a message.

Key terms: local view, global view, state of certainty, state of uncertainty

4.2.1 Before sending the request

4.2.2 After sending the request and before receiving a response

After sending a request but before receiving a response, the sender cannot tell whether the receiver has processed it or what will happen next.

• Uncertainty while awaiting a response: Explain why a sender awaiting a response cannot determine whether the work has been done, is still pending, or will never be done, and describe the different events that could produce that uncertainty.
  Why it matters: The same lack of response can correspond to very different outcomes for the requested work.

Key terms: sender, receiver, request, response

4.2.3 After receiving a response

A received response resolves uncertainty about the work, but TCP’s reliable transmission does not guarantee that an application receives and processes a message.

• Certainty from a response: Explain that, after receiving a response, the sender can treat a reported success as work done and a reported failure as work not done because components do not deceive in this system model.
  Why it matters: A response gives the sender information that it lacked while waiting.
• TCP transmission reliability: Describe how TCP uses acknowledgments, retransmissions, and deduplication to transmit data accurately and completely between sender and receiver, with each side acknowledging data written to a local kernel buffer.
  Why it matters: TCP’s reliability claim is easier to interpret when you know what it acknowledges.
• Transport and application delivery gap: Explain why TCP’s at-least-once delivery at a lower layer does not imply at-least-once delivery to an application: the kernel can acknowledge receipt before delivery to the application, then fail before the message is delivered and processed.
  Why it matters: Reliable transport alone cannot rule out lost application work.

Key terms: response, success, failure, Byzantine failures, TCP, acknowledgments, retransmissions, deduplicates, local buffer, socket interface

4.3 Silence and chatter

The section explains why a missing response cannot reveal whether a request was processed, and why retries and simple duplicate tracking cannot guarantee exactly-once processing.

• Timeouts and uncertain outcomes: Explain why a timeout leaves the sender uncertain whether a request caused a side effect, even though the timeout can signal a possible failure.
  Why it matters: A missing response is not proof that the receiver did nothing.
• Retry and move-on tradeoff: Distinguish the sender's two choices after no response: move on, intending at-most-once delivery and processing, or retry, intending at-least-once delivery and processing. Explain why possible network duplication or continuing loss defeats those intended guarantees.
  Why it matters: Choosing whether to retry changes the risk, but does not by itself ensure exactly-once processing.
• Duplicate-tracking crash windows: Trace why recording a request before processing can lose its processing, while processing before recording can repeat it after a crash and retry.
  Why it matters: Simply keeping a list of seen requests does not close the gap between processing a request and recording it.

Key terms: timeout, partially synchronous, true positive, false positive, side effect, at-most-once processing, at-least-once processing, exactly-once processing, duplicate requests, duplicate

4.4 Exactly-once processing semantics

This section defines exactly-once processing as an application-defined equivalence of outcomes and explains why it requires both sender and receiver participation.

• Equivalent processing outcomes: Explain that exactly-once processing semantics concern the resulting state, not whether a message was physically processed only once. Describe why processing it multiple times may produce an equivalent but unequal state, and why the application determines what counts as equivalent.
  Why it matters: This distinction lets you reason about duplicate processing without requiring every observable detail to be identical.
• End-to-end delivery responsibilities: Explain that exactly-once processing semantics are an end-to-end guarantee requiring at-least-once delivery and processing: the sender keeps submitting requests until it receives a response, and the receiver does not skip processing.
  Why it matters: The desired outcome cannot be guaranteed by the sender or receiver acting alone.

Key terms: exactly-once processing semantics, equivalence, processing outcome, end-to-end guarantee, at-least-once delivery and processing, sender, receiver

4.5 Idempotence

This section explains how idempotence makes repeated message processing have the effect of processing once, and how applications can achieve that effect with idempotence keys and transactional deduplication.

• Idempotent processing effect: Explain that an idempotent operation has the same effect when executed multiple times as when executed once, so at-least-once message processing can provide exactly-once processing semantics when the processing operation is idempotent.
  Why it matters: This is the distinction between preventing retries and making retries safe.
• Inherent operation idempotence: Distinguish the section's write-once register, whose value cannot change after the first set, from a credit-card charge, which can be applied twice if a request is duplicated.
  Why it matters: Whether a retry is safe depends on the operation's actual effect.
• Shared idempotence keys: Describe how the sender attaches the same universally unique idempotence key to a request and all its retries, and the receiver uses that key to deduplicate them. Explain that this is application-specific and requires both ends to participate.
  Why it matters: The receiver needs a way to recognize retries as the same request, not new work.
• Transactional duplicate suppression: Explain the receiver-side pattern of recording an idempotence key and performing the request's side effect transactionally, committing only when a new key entry satisfies a uniqueness constraint.
  Why it matters: Recording the key together with the effect prevents a retry from being treated as new work.

Key terms: idempotence, at-least-once message processing, exactly-once processing semantics, write-once register, credit card, inherently idempotent, idempotence key, UUID, deduplicate, end-to-end guarantee

4.6 Case study: Charging a credit card

The credit-card charging case shows how retries, idempotency keys, and the user’s experience complicate the promise of exactly-once processing.

• Uncertain charge outcomes: Explain why a missing response during airline checkout does not tell the app whether the card was charged, and describe what happens if bounded retries still produce no response.
  Why it matters: A missing response creates uncertainty about the purchase, not proof that the charge failed.
• Reservation-based idempotency keys: Describe how a payment processor’s idempotency key can make repeated charge requests return the first request’s result, and explain how a UUID reservation number could supply the key.
  Why it matters: A stable key lets retries refer to the same purchase rather than create separate charges.
• User-visible exactly-once experience: Distinguish an eventual single effective charge from a checkout experience that feels like one charge to the customer, using the airline app’s multiple pending charges as the example.
  Why it matters: An implementation can meet its own definition of exactly once while still alarming its users.

Key terms: request, response, bounded number of times, idempotency key, reservation number, UUID, status code, body, exactly-once processing semantics, pending charges

4.7 Summary

The section summarizes why failures rule out exactly-once message delivery and processing, how exactly-once processing semantics can still be achieved, and why idempotence is difficult.