chapter five

5 Delivery Semantics

 

This chapter covers:

  • Delivery semantics and their impact
  • Three delivery semantics
  • 1. At-most-once delivery semantic
  • 2. At-least-once delivery semantic
  • 3. Exactly-once delivery semantic

“There’s never enough time to do it right, but there’s always enough time to do it over.”

Jack Bergman

Computers are pretty good at performing accurate calculations. However, when computers working together in a distributed system like many streaming systems, accuracy becomes a little bit more (I mean, a lot more) complicated. Sometimes, we may not want 100% accuracy, because other more important requirements need to be met. “Why do we want wrong answers?”, you may ask. This is a great question and it is the one that we need to ask when designing a streaming system. In this chapter, we are going to discuss an important topic related to accuracy in streaming systems: delivery semantics.

5.1   The latency requirement of the fraud detection system

In the previous chapter, Tim and Tracy have built a credit card fraud detection system which can make a decision within 20 milliseconds for each transaction and store the result into the database.

Now, let’s ask an important question when building any distributed system: what if any failure happens?

5.2   Revisit the fraud detection job

5.3   About accuracy

5.4   Partial result

5.5   A new streaming job to monitor system usage

5.6   The new system usage job

5.7   The requirements of the new system usage job

5.8   New concepts: (number of) times delivered and times processed

5.9   New concept: delivery semantics

5.10   Choosing the right semantics

5.11   At-most-once

5.12   The fraud detection job

5.12.1   The goods

5.12.2   The bads

5.12.3   The hope

5.13   At-least-once

5.14   At least once with acknowledging

5.15   Track events

5.16   Handle event processing failures

5.17   Track early out events

5.18   Acknowledging code in components

5.19   New concept: checkpointing

5.20   New concept: state

5.21   Checkpointing in the system usage job for at-least-once semantic

5.22   Checkpointing and state manipulation functions