Fault isolation contains a component failure so the failed component can be terminated, replaced, or restarted without automatically stopping unrelated work. The practical goal is to keep recovery logic out of normal processing: a log-processing pipeline should not need database-reconnection code tangled into its file and line handling. section 5.1.1 section 5.1.2
When a database connection fails
Consider a FileWatcher that sends logs to a LogProcessor, which uses a Writer and database connection. In a direct object graph, many objects may hold references to the Writer or connection. Replacing the broken connection safely requires preventing concurrent use, adding indirection, and ensuring every reference stops targeting the old instance. An exception also travels up a call stack, where recovery may lack the context needed to select redundancy or continue with the right input. section 5.1.1
Figure 5.1 Processing logs from the FileWatcher, to the LogProcessor, to the Writer, and to the database
This pipeline makes the containment problem visible: a failure in the Writer or database connection can propagate through directly coupled processing components. section 5.1.1
Actor supervision changes the boundary. A supervisor wraps an actor’s behavior and decides what happens after an exception. It can terminate the actor, restart it, or replace it with a fresh actor instance. Other actors keep actor references rather than direct references to a particular instance, so replacement can occur without requiring those actors to be rewritten around the new instance. section 5.1.2
Figure 5.6 Supervisor wrapping an actor’s behavior and deciding on a recovery action after an exception
The supervisor boundary separates message processing from the recovery decision made after an actor throws an exception. section 5.1.2
A let-it-crash design supports fault isolation through distinct mechanisms:
- Terminate the faulty actor, removing it from the ActorSystem when continuing it is unsafe. section 5.1.2
- Suspend the crashed actor’s mailbox while the supervisor decides the recovery action. section 5.1.2
- Re-create an actor from its Behavior, allowing a fresh instance without exposing reconstruction details to the supervisor. section 5.1.2
- Use restart to return an actor to an initial state when that is the appropriate recovery action. section 5.1.2
- Keep ordinary message handling separate from supervision and fault-recovery flows so each can evolve independently. section 5.1.2
Avoiding a poisoned mailbox
A LogProcessor may fail on a corrupted file or an unparseable line. Blindly reprocessing that message can repeatedly trigger the same failure and prevent every later message from being processed: a poisoned mailbox. After an exception, the failed message is not normally left in the mailbox. Reprocess only when it is clear that the message did not cause the error. section 5.1.2
Place actors most likely to crash low in the actor hierarchy. That placement limits the scope of a fault and leaves higher-level components available to supervise, replace, route around, or otherwise decide the response. A replacement actor may establish a fresh database connection, potentially to a different database, while surrounding actors remain unaffected. section 5.1.2
