Chapter 11: Durable executions

Introduction

The introduction previews the chapter’s path from process duration and failure-tolerant execution to sagas and durable executions, presenting durable execution as an abstraction that conceals failure.

• Durable execution analogy: Explain that durable executions are an emerging abstraction for distributed systems that conceals the possibility of failure, much as transactions do for databases.
  Why it matters: This analogy frames why the chapter treats durable execution as an abstraction rather than just a recovery technique.
• Chapter topic progression: Describe the chapter’s announced progression: short-running versus long-running processes, failure-free definitions, failure-tolerant executions, and sagas versus durable executions.
  Why it matters: The progression shows which distinctions the chapter will build before comparing its two approaches.

Key terms: durable executions, transactions, abstraction, short-running, long-running, failure-free definitions, failure-tolerant executions, sagas

11.1 The pitfalls of partial executions

The section uses a signup failure to show why sequential atomic actions can leave a process partially executed, then introduces failure-transparent recovery and two meanings of atomicity.

• Partial signup execution: Explain how a crash between charging a user's card and granting content access leaves the signup process partially executed, even if each step is atomic.
  Why it matters: A concrete partial execution shows why making individual steps atomic does not make the whole process atomic.
• Failure-transparent execution: Describe the ideal of failure-transparent execution: after a failure and recovery, the process is observably equivalent to having completed or to having done nothing, rather than exposing only a completed prefix.
  Why it matters: This gives a precise target for recovering from partial executions.
• Two flavors of atomicity: Distinguish concurrency atomicity, which hides intermediary states from other processes, from failure atomicity, which requires an all-or-nothing outcome after failure even though intermediary states may be observable. Explain why failure atomicity matters for moving from one correct state to another.
  Why it matters: The distinction identifies which atomicity property addresses a crash that leaves lasting partial effects.

Key terms: partial execution, sequential composition, atomic actions, crash failure, failure-transparent, prefix, recovered execution, concurrency atomicity, failure atomicity, isolation

11.2 System model

This section separates code definitions from running executions so failure-related terms can be applied precisely.

• Definitions and executions: Distinguish a process definition—the code—from an execution of that code, and state which failure-related terms describe each.
  Why it matters: Keeping code and its execution separate makes it easier to reason clearly about what happens when failures occur.

Key terms: definition, execution, failure-aware, failure-agnostic, failure-free, failure-tolerant

11.2.1 Process definition

Defines a process as a sequence of failure-atomic actions ending in successful termination.

• Recursive process definition: Explain that a process definition P consists of failure-atomic steps and can be written as P = A • P’ | ε: it either performs action A and continues as P’, or terminates successfully.
  Why it matters: This definition provides the basic unit for discussing how a process executes and recovers from failure.

Key terms: process definition, failure-atomic action, successor process, ε

11.2.2 Process execution

This section models a process execution as a trace of observable steps and shows why failure atomicity does not carry over from individual steps to a multistep process.

• Process execution traces: Describe a process execution as a trace in which each step of the process definition appears as an observable event, and a crash-stop failure may end the trace before completion.
  Why it matters: The trace makes it possible to reason precisely about what happened before a failure.
• Failure atomicity across steps: Explain that a single-step process execution inherits its step's failure atomicity, while a multistep execution does not: failure atomicity does not compose.
  Why it matters: Individually atomic steps can still leave a partially completed process.
• Logical execution length: Distinguish short-running from long-running executions by their number of steps: one versus multiple. Explain that these terms describe logical time rather than elapsed physical time.
  Why it matters: This distinction describes the structure of an execution, not how long a clock says it took.

Key terms: process definition, process execution, trace, observable event, Crash-Stop, ✓, failure atomicity, single step, multiple steps, failure atomicity does not compose

11.3 The concept of failure-transparent recovery

Defines failure-transparent recovery through application-specific equivalence between failure-free and recovered executions, then illustrates how much replay that equivalence may allow.

• Failure-transparent recovery criterion: Explain that recovery is failure-transparent when a failed, recovered, and continued execution produces events equivalent to a failure-free execution. State that transient failures need not prevent completion, whereas permanent failures may, and that this section considers forward recovery.
  Why it matters: This criterion says what recovery must preserve without requiring failures to be invisible in every detail.
• Application-specific event equivalence: Explain that an application-specific equivalence function decides whether the event sequences of a failure-free run and a recovered run count as equivalent; engineers choose what differences are acceptable for their application.
  Why it matters: The equivalence rule determines whether replayed events are acceptable or change the result that matters.
• Replay equivalence variants: Distinguish the section's equivalence rules using the a-to-f printing example: identity permits no repeated events, while other rules permit replay of the last event, the last n events, or the entire process from the start. Explain that identity amounts to exactly-once processing, which is not always achievable.
  Why it matters: These rules make the possible effects of recovery concrete, from no visible replay to a complete restart.

Key terms: failure-transparent recovery, failure-free execution, forward recovery, equivalence function, sequence of events, identity function, exactly-once processing, valid to duplicate last event equivalence function, valid to duplicate last n events equivalence function, restart equivalence function

11.4 Strategies of failure-transparent recovery

The section introduces idempotence as a safeguard against duplicated actions and names restart and resume as two patterns for continuing an interrupted process.

• Idempotent actions under duplication: Explain that an action is idempotent when doing it multiple times has the same result as doing it once, and that idempotent actions help a system behave correctly if events are duplicated.
  Why it matters: Recovery may encounter duplicated events, so repeated actions must not change the intended result.
• Interrupted execution recovery patterns: State that when failure interrupts a process execution, the system must recover and continue it, and identify restart and resume as the two prevalent patterns introduced here.
  Why it matters: These two patterns organize the section’s account of how an interrupted execution can continue.

Key terms: idempotence, event duplication, process execution, restart, resume

11.4.1 Restart

Restarting a failed process execution is simple when its steps are idempotent, but restarting from the beginning can change intended timing or nondeterministic outcomes.

• Restart from the beginning: Explain that restart recovers from a failure by running the process execution again from the beginning, and that it is simple to implement when the process steps are idempotent.
  Why it matters: This establishes both the appeal of restart and the condition that makes repeating steps straightforward.
• Delayed payment restart problem: Describe why restarting the signup process is insufficient when account creation should happen now but payment collection should wait one week: restarting the entire process resets the delay.
  Why it matters: Repeating a process from its beginning may fail to preserve the timing the process intended.
• Nondeterministic action risk: Explain that restarting a process containing nondeterministic actions, such as reading the current time or generating a random number, may produce a different outcome and thus incorrect behavior.
  Why it matters: A restarted run may not reproduce the choices or values used in the original run.

Key terms: restart, process execution, idempotent, user signup, create the account, one-week delay, collecting payment, nondeterministic actions, current time, random number

11.4.2 Resume

The section explains how a process execution can recover from failure by resuming from its most recent save point.

• Save point recovery: Explain that a failed process execution restarts from its most recent save point and effectively resumes because sufficient information was captured between process steps.
  Why it matters: Save points let a process continue after failure without starting its execution over from the beginning.

Key terms: process execution, save point, resume

11.5 Implementation of failure-transparent recovery

This section introduces application-level and platform-level ways to implement failure-transparent recovery, distinguished by where failure handling lives.

• Recovery implementation levels: Distinguish application-level from platform-level failure-transparent recovery: application-level recovery makes process definitions explicitly handle potential failures, while platform-level recovery keeps those definitions failure-agnostic by handling failures within the platform.
  Why it matters: The distinction explains whether developers must account for failures in each process definition or can rely on the platform to do so.

Key terms: application-level recovery, platform-level recovery, failure-aware, failure-agnostic, process definitions

11.5.1 Application-level implementation: Sagas

This section explains how sagas provide failure-transparent executions through application-level failure handling, and why that approach can obscure business logic.

• Application-level failure responsibility: Explain that an application-level implementation makes the developer responsible for detecting and mitigating failures, so the process definition explicitly accounts for them; a saga is such a failure-aware definition intended to produce failure-transparent executions.
  Why it matters: This distinguishes a saga's failure-aware code from the failure-transparent execution it provides.
• Persisted saga progress: Describe the common state-machine-based saga approach: after each step, it persists the saga's state and the trigger for the next step, enabling eventual execution. State that the section presents this as a common approach, not a canonical one.
  Why it matters: Persisted progress explains how this approach can continue a process despite interruption.
• Business-logic clarity tradeoff: Explain the author's view that explicitly accounting for failure in a saga can obscure the signup process's core business logic and make it harder to understand and maintain. Contrast this with durable executions, whose definitions need not explicitly account for failure.
  Why it matters: The contrast motivates the platform-level approach discussed next.

Key terms: sagas, failure-aware process definitions, failure-transparent process executions, failure detection, failure mitigation, state-machine-based implementations, state of the saga, trigger for the next step, persisted, eventual execution

11.5.2 Platform-level implementation: Durable execution

The section explains how platform-level durable execution hides failure handling from process code and compares log-based replay with state-based resumption.

• Failure-agnostic process definitions: Explain that platform-level failure transparency lets developers write process definitions without coding failure detection or mitigation, while executions can still recover transparently.
  Why it matters: It separates what the developer writes from the recovery work the platform performs.
• Log-based replay and limits: Trace how a log-based run time durably records each step's output, then restarts after failure and reuses logged results to fast-forward past previously executed steps. Explain why replay requires deterministic execution and why log growth can become a limit.
  Why it matters: Replay avoids repeating completed work, but its constraints affect which processes can use it.
• State-based continuation and tradeoff: Describe how a state-based run time saves the execution's continuation after each step and restores it after failure to resume without replay. Contrast its lack of a determinism or growing-log requirement with its need for run-time support for serializable continuations.
  Why it matters: Saving the continuation removes replay's constraints but depends on support that most run times do not yet provide.

Key terms: failure-agnostic, failure-oblivious, failure-transparent, durable executions, log-based implementation, durable log, replay, deduplicates, deterministic execution, state-based implementation

11.6 Summary

The section recaps failure transparency, the application- and platform-level ways to achieve it, and two durable-execution strategies.