Fault tolerance is the ability to keep important work available when components fail. It does not mean preventing every fault: networks, third-party services, and servers can be unavailable. Instead, anticipate recovery requirements and prevent one failure from becoming a total crash. section 5.1 section 5.6
| Strategy |
Description |
| Fault containment or isolation |
A fault should be contained within a part of the system and not escalate to a total crash. |
| Structure |
Isolating a faulty component means a structure must exist to isolate the component from the rest of the system. The system needs a defined structure in which active parts can be isolated. |
| Redundancy |
A backup component should be able to take over when a component fails. |
| Replacement |
If a faulty component can be isolated, you can replace it in the structure. The other parts of the system should be able to communicate with the replaced component just as they did with the failed component. |
| Reboot |
If a component gets into an incorrect state, you need the ability to get it back to a defined initial state. The incorrect state may be the reason for the fault, and it may not be possible to predict all the incorrect states the component can get into because of dependencies outside of your control. |
| Component lifecycle |
A faulty component needs to be isolated. If it can’t recover, it should be terminated and removed from the system or reinitialized with a correct starting state. A defined lifecycle must exist to start, restart, and terminate the component. |
| Suspend |
When a component fails, all calls to the component should be suspended until the component is fixed or replaced so the new component can continue the work without dropping a beat. The call handled at the time of failure also should not disappear—it could be critical to recovery. Further, it may contain information critical to understanding why the component failed. You may want to retry the call when you’re sure there was another reason for the fault. |
| Separation of concerns |
It’s great if the fault-recovery code can be separated from the normal processing code. Fault recovery is a cross-cutting concern in the normal flow. A clear separation between normal flow and recovery flow simplifies the work that needs to be done. Changing the way the application recovers from faults is simpler if you achieve this clean separation. |
These strategies turn failure handling into system structure: isolate the fault, provide a backup, replace failed parts, or return incorrect state to a defined initial state. section 5.1
Direct object calls and exception propagation make recovery difficult: processing and recovery logic become tangled, input may need buffering for retries, and concurrent callers add locking concerns. Actor references and recreatable behaviors instead support isolating components and separating functional work from fault recovery. section 5.1.1 section 5.1.1
A fault-tolerant design needs explicit choices about what happens after a failure:
- Contain or isolate the faulty part so the failure does not escalate through the system. section 5.1
- Use redundancy so a backup component can take over. section 5.1
- Replace an isolated component while preserving how other parts communicate with it. section 5.1
- Resume, stop, or restart an actor through supervision; stopping is the default strategy. section 5.6
Recovering a clustered text-processing job
When all aggregator nodes are stopped, the master records failed texts as lag rather than losing them. After a new aggregator joins and workers register, delegation resumes and the lag falls to zero. Message ordering remains indeterminate, so the accumulated lag is not ordered. section 8.5 section 8.5
Local supervision lets the remaining message-based actors continue when one actor goes away, but it does not protect against catastrophic failure. Clustering completes the fault-tolerance story by allowing work to resume after failed nodes are replaced. section 5.6 section 8.5
