liveBook · Manning

Andon

As explained in Build High Value Software Systems

A visual management system that signals abnormal conditions so work can be stopped or corrected.

An alert is useful only if someone—or something—can act on it. Andon pairs a visible signal of abnormal behavior with a way to stop the damage. In software, stopping the line may mean halting a rollout, freezing deployments, disabling a feature, or isolating a failing request rather than shutting down the whole service. section 12.3.1 section 4.2.2

The signal should point toward an actionable problem. A specification can expose code that has drifted from intent; support tickets or unexpected usage can reveal a feature that behaves badly in practice. At an integration boundary, a dead letter queue preserves invalid messages for investigation while keeping them out of the main queue. A growing queue signals a possible upstream defect. section 4.2.2 section 11.7

The response can operate at three levels:

  • Signal only: alert an on-call engineer, who investigates and chooses an action. Recovery waits on human diagnosis and access. section 12.3.3
  • Signal and stop: automatically contain the problem, then alert a human. A circuit breaker stops calls to a failing service and reports its open state. section 12.3.3 section 11.7
  • Signal, stop, and recover: contain the problem and attempt a known remedy, such as rolling back a deployment after elevated error rates. section 12.3.3

Make the stop practical

Suppose a new export format causes unexpected behavior. A configuration switch that disables its initialization gives the team an immediate response; a module that can later be removed without widespread edits keeps that response safe. For a deployment, an error-rate spike can instead trigger an automatic rollback. Both approaches close the gap between noticing trouble and acting on it. section 4.2.2 section 12.3.3

Listing 12.3 Automatic rollback on elevated error rate (Level 3 andon)
def post_deploy_monitor(deploy_id: str, baseline_error_rate: float):
    """Watch error rate for 10 minutes after deploy. Rollback if it spikes."""
    for minute in range(10):
        current_rate = get_error_rate(window_minutes=1)
        if current_rate > baseline_error_rate * 3:
            trigger_rollback(deploy_id)
            notify_team(
                f"Auto-rollback: deploy {deploy_id}. "
                f"Error rate {current_rate:.2%} vs baseline {baseline_error_rate:.2%}"
            )
            return "rolled_back"
        time.sleep(60)
    return "stable"

The rollback listing watches post-deployment error rates, rolls back when they exceed a defined threshold, and notifies the team. section 12.3.3

Choose automation by confidence in the diagnosis. If the on-call engineer would take the same action every time, automate it. If a symptom such as high latency has several plausible causes, contain what you safely can and leave the remedy to a human. Before shipping, ask whether the behavior can be disabled, bypassed, or removed without destabilizing the system; a signal without a usable stop is not enough. section 12.3.3 section 4.2.2

How Build High Value Software Systems covers it

How deep it goes
Defines signal plus stop, distinguishes three intervention levels, and gives criteria for automating recovery.
Examples
Uses a switchable export module, accumulated dead-letter messages, and post-deployment error-rate rollback as concrete cases.
What it assumes
Assumes familiarity with deployments and on-call operations while defining the signal-and-stop principle directly.
What you can do after
Operate/configure it by pairing actionable signals with safe stops and automating recovery only when the remedy is clear.
How it connects
Connects specifications, dead letter queues, circuit breakers, feature flags, and automatic rollback to detection and intervention.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build High Value Software Systems. The section links point to the parts of the book it draws on.