Resilience is the ability to recover from a failure or disturbance and keep operating acceptably. In a distributed system, it means expecting small failures and preventing them from bringing down the whole application. section 10.2 section 2.4
The resilient waiter
A restaurant runs out of steak. Instead of leaving the customer waiting indefinitely, the waiter waits briefly, then offers chicken. The fallback does not fix the missing steak; it keeps the meal moving with a reduced choice. section 10.2
Software needs similar shock absorbers. A timeout limits waiting, a retry gives a temporary problem another chance, and a fallback offers an alternative when the original request cannot succeed. Circuit breakers are another protective pattern for failing dependencies. section 10.2 section 10.1
Choose the response based on the failure:
- For transient glitches, such as a network blip, timeout, or HTTP 503, retries with exponential backoff can give the dependency time to recover. section 10.1
- For non-transient faults, such as invalid requests, authentication failures, or corrupted messages, repeated attempts waste resources. Fail fast or route a poison message to a dead letter queue. section 10.1
Keep failure handling separate from business logic. Charging a card is a business concern; surviving a network failure is an infrastructure concern. A resilience library can wrap a fragile call without tangling retry loops and sleep commands into the function itself. section 10.4
Listing 10.1 The Fragile Way
import requests
def charge_credit_card(amount): #A
print("Attempting to charge card...")
response = requests.post("https://api.FlakyPayments.com/charge",
json={"amount": amount, "order_id": "12345"},
)
response.raise_for_status() # Throws an error if the API is down
return response.json()This payment call illustrates the fragile starting point: an API failure raises an error without a protective resilience wrapper. section 10.4
Resilience also requires operational discipline. Timeouts, fallbacks, and circuit breakers account for imperfect systems, but they do not excuse leaving production functionality running solely for convenience. Active code, threads, connections, and background tasks can all become failure risks. section 10.7.3
















