Chapter 3: Failure tolerance

Introduction

The chapter introduction defines the purpose of failure tolerance and frames the chapter as a theoretical and practical exploration of failure.

• Purpose of failure tolerance: Explain that the primary reason to think about failure is to ensure failure tolerance: a guarantee that a distributed system functions in a well-defined manner even when failures occur.
  Why it matters: This purpose connects the chapter’s different ways of thinking about failure.
• Theoretical and practical perspectives: Describe how the chapter divides the broad topic of failure into a theoretical perspective followed by a practical perspective, with failure detection and mitigation among the topics previewed.
  Why it matters: The two perspectives give you a way to organize the chapter’s discussion.

Key terms: failure tolerance, well-defined manner, theoretical, practical, failure detection, failure mitigation

3.1 In theory

This section defines failure tolerance through system states and transitions, introduces failure recovery, and sets a boundary around failures a system cannot tolerate.

• System states and transitions: Describe the legal, illegal, and intolerable states and the distinction between normal and failure transitions. Explain that a failure is an unwanted but possible transition from a good state to a bad state.
  Why it matters: States and transitions give you a precise way to describe what has gone wrong.
• Failure tolerance and recovery: Explain that failure tolerance means behaving in a well-defined manner while in a bad state, and that failure recovery is a sequence of transitions from an illegal state back to a legal state. Distinguish both from an intolerable situation, which the framework does not attempt to tolerate.
  Why it matters: A system can have a defined response to failure even before it has recovered.
• Failure terminology choice: Explain why the book consistently uses “failure” and “failure tolerance” rather than relying on distinctions among fault, error, and failure.
  Why it matters: Using the book’s terminology avoids reading a disputed distinction into its explanations.

Key terms: legal state, illegal state, intolerable state, normal transition, failure transition, failure tolerance, failure recovery, fault, error, failure

3.2 Types of failure tolerance

This section introduces safety and liveness as the properties used to describe failure tolerance, noting that failures may force a compromise between them.

• Safety and liveness guarantees: Distinguish safety, which guarantees that something bad never happens, from liveness, which guarantees that something good eventually happens.
  Why it matters: These two guarantees give you a way to describe what a system preserves when failures occur.
• Failure tolerance tradeoffs: Explain that failure tolerance can be defined by which safety and liveness properties a system guarantees: without failure it guarantees both, while with failure it may have to compromise.
  Why it matters: This framing lets you describe failure tolerance in terms of guarantees rather than simply saying a system survives failures.

Key terms: safety, liveness, failure tolerance

3.2.1 Masking failure tolerance

This section defines masking failure tolerance as preserving both safety and liveness despite failures, while noting that it may be too costly or impossible to guarantee.

• Masking failure tolerance: Explain that a system provides masking failure tolerance when it guarantees both safety and liveness in the presence of failure, making failures effectively transparent. Describe why this desirable guarantee may be too costly or impossible in some scenarios.
  Why it matters: This distinction helps you reason about what a system can promise when failures occur.

Key terms: masking failure tolerance, safety, liveness, failure transparency, CAP conjecture

3.2.2 Nonmasking failure tolerance

Defines nonmasking failure tolerance as continued progress during failure without a guarantee that the system will avoid mistakes.

• Nonmasking failure tolerance: Explain that nonmasking failure tolerance guarantees liveness, but not safety, in the presence of failure. Describe how a normally first-in, first-out queue can keep delivering messages during a failure while temporarily delivering them out of order.
  Why it matters: This distinction helps you tell continued service apart from continued correctness during a failure.

Key terms: nonmasking failure tolerance, liveness, safety, first in, first out

3.2.3 Fail-safe failure tolerance

Defines fail-safe failure tolerance as preserving safety during failures even when progress stops, using a message queue as an example.

• Fail-safe safety and progress: Explain that fail-safe failure tolerance guarantees safety, but not liveness, in the presence of failure. Describe how a queue may stop delivering messages during a failure rather than risk delivering them out of order.
  Why it matters: A system can avoid incorrect behavior during a failure without continuing to make progress.

Key terms: fail-safe failure tolerance, safety, liveness, queue

3.2.4 None of the above

The section uses a four-state process to distinguish a recoverable safety violation from a failure state that offers no guarantee of safety or liveness.

• Failure-tolerance boundary: Explain that a system is not failure-tolerant if it guarantees neither safety nor liveness in the presence of failure.
  Why it matters: This sets the boundary between tolerating a failure and merely entering an uncontrolled state.
• Four-state process example: Describe the stipulated safety and liveness properties of process P, then distinguish x = 0, which is unsafe but still live, from the intolerable x = 3 state, which has no guaranteed return to another state.
  Why it matters: The example shows why an unsafe state can still count as nonmasking failure tolerance while another state cannot.

Key terms: safety, liveness, failure-tolerant, process P, x, nonmasking failure-tolerant, intolerable

3.3 In practice

This introduction shifts from theoretical failure tolerance to a practical failure-handling strategy for service-oriented distributed systems and sets aside detailed discussion of retries for chapter 4.

• Practical failure-handling focus: Explain that the section moves from theoretical foundations about failures toward a practical foundation and an outline for handling failures in a service-oriented distributed system.
  Why it matters: This shift frames the practical topics that follow.
• Retries deferred to chapter four: State that retries are common for mitigating failures in (micro)services-oriented distributed systems, but their details are deferred to chapter 4's discussion of message delivery and processing semantics.
  Why it matters: It clarifies why this section does not explore a common failure-mitigation technique in depth.

Key terms: failure-handling strategy, service-oriented distributed system, retry mechanisms, mitigation, message delivery and processing semantics

3.3.1 System model

This section models service orchestration as a sequence of requests and explains why failures must not leave a process partially applied.

• Service orchestration process model: Describe a service consumer executing a process whose steps are requests to one or more service providers, with components communicating through request–response messages.
  Why it matters: This model gives the rest of the chapter a concrete setting for discussing failures.
• Complete versus partial application: Distinguish successful total application, which completes every step and moves between correct states, from failure-induced partial application, which may leave the system incorrect. Explain that recovery should make the result observably equivalent either to completing the process or to not applying it.
  Why it matters: The desired end states define what it means to handle a failed process correctly.
• Checkout charge outcome: Apply the completeness requirement to checkout: a successful checkout charges the credit card exactly once, while a failed checkout does not charge it; a charge must not remain when goods are not shipped or tickets are not issued.
  Why it matters: The checkout example makes the cost of partial application tangible.

Key terms: service orchestration, service consumer, service provider, process, request–response, total application, partial application, completeness, forward recovery, backward recovery

3.3.2 Failure handling

Failure handling detects unwanted events and mitigates them, using detection to protect safety and mitigation to restore liveness.

• Failure detection and safety: Explain that failure handling begins by detecting an unwanted event, and that detection can protect safety by prompting the system to inhibit dangerous actions.
  Why it matters: Detecting a failure matters because the system may need to stop unsafe actions before it can recover.
• Failure mitigation and liveness: Explain that mitigation follows detection and can restore liveness by addressing the failure so the system resumes making progress. Describe the section's example in which a node rejoins a quorum and resumes responding to requests.
  Why it matters: Mitigation matters because preserving safety alone does not get a stalled system moving again.

Key terms: failure handling, failure detection, safety, failure mitigation, liveness, quorum

3.3.3 Failure classification

This section classifies failures by the layer that can handle them and by how likely they are to recur, to guide detection and mitigation.

• Layer-relative system roles: Explain how a layered system makes downcalls and occasional upcalls, and why application and platform are roles relative to the layer being considered rather than fixed names for particular components.
  Why it matters: The layer you are reasoning from determines what counts as an application or platform failure.
• Spatial failure classification: Explain how the spatial dimension classifies a failure as application-level or platform-level according to the lowest layer that can detect and mitigate it correctly and completely. Contrast InsufficientFunds with CouldNotConnect.
  Why it matters: Placing failure handling at a layer that cannot interpret or fully resolve the problem will not handle it correctly.
• Temporal failure patterns: Distinguish transient, intermittent, and permanent failures by the probability of another failure after one has occurred and by whether the problem repairs itself. Explain how different causes can give the same CouldNotConnect message different temporal classifications.
  Why it matters: The likely recurrence and repair behavior of a failure affect how it should be handled.

Key terms: layered architecture, downcall, upcall, application layer, platform layer, infrastructure layer, spatial dimension, end-to-end argument, application-level failure, platform-level failure

3.3.4 Failure detection

The section broadens failure detection beyond component crashes, then explains how message-based monitoring works and why its conclusions can be uncertain.

• Failures and their witnesses: Explain that a failure detector can be any witness or predicate confirming an unwanted state transition that leaves the system in an illegal state, not just a mechanism for spotting crashed components. Describe how a step’s return code can reveal an application-level or platform-level failure.
  Why it matters: This broader view helps you recognize failures that a component-health check would miss.
• Component monitoring and timeouts: Describe pull requests and push heartbeats as ways an observer checks an observed component at set intervals. Explain that when an expected message does not arrive in time, a timeout serves as the witness for a suspected crash.
  Why it matters: Timeouts are a practical way to detect complete unavailability, but they produce a suspicion rather than always proving a crash.
• Detection guarantees and uncertainty: Distinguish completeness, which means eventually detecting every actual failure, from accuracy, which means every reported failure is real. Explain why a missing message proves failure in the stated synchronous, reliable-network case but cannot support both guarantees in a partially synchronous or asynchronous system with an unreliable network.
  Why it matters: It prevents a system from treating every missing message as certain proof that another component crashed.
• Limits of component supervision: Explain that monitoring one component raises the further question of who monitors its observer. State that supervision strategies, such as Erlang/OTP supervision trees, can address this in practice, but some failures ultimately require human intervention.
  Why it matters: Monitoring arrangements also have limits and cannot eliminate every need for intervention.

Key terms: failure detector, witness, predicate, application-level failure, platform-level failure, observer, observed component, pull, push, heartbeats

3.3.5 Failure mitigation

This section explains failure mitigation by contrasting backward recovery with forward recovery.

• Backward and forward recovery: Distinguish the two ways failure mitigation resolves a suspected failure: backward recovery returns an intermediary illegal state to the equivalent of the initial legal state, while forward recovery advances it to the equivalent of the final legal state. Explain why the former generally need not repair the underlying cause but the latter generally does, and connect them to compensation and retries.
  Why it matters: The direction of recovery determines whether the system abandons the interrupted process or tries to complete it.

Key terms: failure mitigation, backward recovery, forward recovery, intermediary, illegal state, initial, legal state, final, legal state

3.3.6 Putting everything together

The section combines failure classification and mitigation into an escalating strategy for completing an orchestrated process despite failures.

• Failure ownership by classification: Explain that an unambiguous application-level failure is handled by the application, while an unclassified failure or a clear platform-level failure is handled first by the platform. Describe backward recovery by compensating completed steps as an application-level response.
  Why it matters: Classifying a failure determines who should handle it and what kind of recovery is appropriate.
• Escalating platform recovery steps: Trace the platform’s escalating response: retry immediately for an assumed transient autorepair failure, retry multiple times with backoff for an assumed intermittent autorepair failure, then suspend the process for manual repair and resume it to retry. State that unsuccessful platform mitigation is elevated to the application.
  Why it matters: Escalation gives recoverable failures several chances to resolve before the application must handle them.
• Failed compensation and human intervention: Explain that backward recovery can itself fail, such as when a credit-card refund fails, and that the failure must then be elevated from the application to human operators for detection and mitigation.
  Why it matters: Even compensation is not guaranteed to work, so the strategy needs a final path for human intervention.

Key terms: application-level failure, platform-level failure, backward recovery, compensating, transient, intermittent, autorepair, manual-repair, immediate retry, backoff strategy

3.4 Summary

The section summarizes how failure handling aims for failure tolerance by detecting and mitigating failures according to their classification and the desired tolerance strategy.