An alert is useful only if someone—or something—can act on it. Andon pairs a visible signal of abnormal behavior with a way to stop the damage. In software, stopping the line may mean halting a rollout, freezing deployments, disabling a feature, or isolating a failing request rather than shutting down the whole service. section 12.3.1 section 4.2.2
The signal should point toward an actionable problem. A specification can expose code that has drifted from intent; support tickets or unexpected usage can reveal a feature that behaves badly in practice. At an integration boundary, a dead letter queue preserves invalid messages for investigation while keeping them out of the main queue. A growing queue signals a possible upstream defect. section 4.2.2 section 11.7
The response can operate at three levels:
- Signal only: alert an on-call engineer, who investigates and chooses an action. Recovery waits on human diagnosis and access. section 12.3.3
- Signal and stop: automatically contain the problem, then alert a human. A circuit breaker stops calls to a failing service and reports its open state. section 12.3.3 section 11.7
- Signal, stop, and recover: contain the problem and attempt a known remedy, such as rolling back a deployment after elevated error rates. section 12.3.3
Make the stop practical
Suppose a new export format causes unexpected behavior. A configuration switch that disables its initialization gives the team an immediate response; a module that can later be removed without widespread edits keeps that response safe. For a deployment, an error-rate spike can instead trigger an automatic rollback. Both approaches close the gap between noticing trouble and acting on it. section 4.2.2 section 12.3.3
Listing 12.3 Automatic rollback on elevated error rate (Level 3 andon)
def post_deploy_monitor(deploy_id: str, baseline_error_rate: float):
"""Watch error rate for 10 minutes after deploy. Rollback if it spikes."""
for minute in range(10):
current_rate = get_error_rate(window_minutes=1)
if current_rate > baseline_error_rate * 3:
trigger_rollback(deploy_id)
notify_team(
f"Auto-rollback: deploy {deploy_id}. "
f"Error rate {current_rate:.2%} vs baseline {baseline_error_rate:.2%}"
)
return "rolled_back"
time.sleep(60)
return "stable"The rollback listing watches post-deployment error rates, rolls back when they exceed a defined threshold, and notifies the team. section 12.3.3
Choose automation by confidence in the diagnosis. If the on-call engineer would take the same action every time, automate it. If a symptom such as high latency has several plausible causes, contain what you safely can and leave the remedy to a human. Before shipping, ask whether the behavior can be disabled, bypassed, or removed without destabilizing the system; a signal without a usable stop is not enough. section 12.3.3 section 4.2.2
