liveBook · Manning

Failure transparency

As explained in Think Distributed Systems

Failure-transparent recovery is recovery that hides failures from process definitions and other observers so the system continues to appear to provide its specified behavior.

Failure transparency means that a failed execution, after recovery, appears equivalent to an uninterrupted execution. Transient failures do not prevent completion, although permanent failures may. The comparison depends on an application-defined equivalence function. section 11.3 section 11.6

Separate process definitions from executions: failure-aware and failure-agnostic describe definitions, while failure-free, failure-tolerant, and failure-transparent describe executions. Recovery can therefore be placed in the application or hidden inside the platform. section 11.5 section 11.2

A charge without access

A signup process charges a payment card and then creates an account. If it crashes between those actions, the user is charged without receiving access: sequentially composed atomic actions have produced an incorrect intermediary state. Failure-transparent recovery aims to make the result equivalent to completing the whole process or observing no partial effect. section 11.1

Listing 11.2 Printing letters a to f
function alphabet() {
  print("a");
  print("b");
  print("c");
  print("d");
  print("e");
  print("f");
}

The letter-printing process makes recovery equivalence concrete: a failed run may repeat the last event, several events, or restart from the beginning, depending on the chosen equivalence. section 11.3

Recovery strategies differ in where they handle failure and what they preserve:

  • Application-level recovery makes process definitions failure-aware and requires them to handle possible failures explicitly. section 11.5
  • Platform-level recovery keeps process definitions failure-agnostic and abstracts failure handling from developers. section 11.5
  • Log-based durable execution records step outputs, replays after failure, and deduplicates events; state-based execution restores the continuation without replaying steps. section 11.6

How Think Distributed Systems covers it

How deep it goes
Defines recovery through event-sequence equivalence, then distinguishes application and platform mechanisms while noting permanent-failure limits.
Examples
Uses a payment-and-account signup failure and a letter-printing process to expose partial execution and recovery equivalence.
What it assumes
Assumes basic process execution and failure concepts while establishing the distinction between definitions and executions from first principles.
What you can do after
Evaluate recovery designs by choosing an equivalence function and distinguishing application-level from platform-level failure handling.
How it connects
Connects failure transparency to safety, liveness, atomicity, exactly-once processing, forward recovery, and durable execution.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Think Distributed Systems. The section links point to the parts of the book it draws on.