liveBook · Manning

Rollback

As explained in Designing AI Systems

A rollback is the restoration of a deployed system to a previously known version or configuration after a problematic change.

Rollback restores a deployed system to a previously known version or configuration after a problematic change. In a production deployment, it is the recovery path when the new version’s application-level error rate exceeds the established baseline. section 9.7 section 9.5.1

A production rollback pattern combines:

  • Incremental rollout: Kubernetes replaces pods progressively, keeping old and new versions active during the transition. section 9.7
  • Application-level monitoring: the Workflow Service compares new-version error rates with the old version’s baseline rather than relying only on container health checks. section 9.7
  • Versioned deployment state: each deployment increments a workflow version, allowing the service to identify and restore the previous version. section 9.5.1
  • Explicit service support: deployment operations include status checking and a RollbackWorkflow operation for the deploy CLI. section 9.5

Automated deployment rollback

A developer deploys an updated container image. Kubernetes performs a rolling update while the Workflow Service watches error rates through the Observability Service. If errors rise significantly above the previous baseline, the service reverts the Deployment to its previous container image. The same progressive-enforcement principle applies to new policies: shadow evaluation and canary traffic reduce blast radius before full enforcement, with automatic rollback available when the change causes widespread failures. section 9.7 section 6.9

Figure 9.4 The three consumers of the Workflow Service contract. The deploy CLI connects to registry operations (RegisterWorkflow, UpdateWorkflow, DeleteWorkflow) and deployment operations (DeployWorkflow, RollbackWorkflow). The SDK's runtime server inside workflow containers connects to job operations (CreateJob, UpdateJobProgress, CompleteJob, SaveJobCheckpoint , FailJob). External clients connect to job operations (GetJobStatus, CancelJob) and optionally to deployment status (GetDeploymentStatus).

Inspect the Workflow Service consumers and operations that connect deployment status, DeployWorkflow, and RollbackWorkflow. section 9.5

How Designing AI Systems covers it

How deep it goes
Explains rolling updates, metric-based decisions, version tracking, and service operations, but does not detail rollback data-plane mechanics.
Examples
Uses container-image replacement after elevated error rates and progressive policy enforcement with shadow and canary stages.
What it assumes
Assumes Kubernetes deployments, container images, application error metrics, and a Workflow Service that tracks deployment versions.
What you can do after
Operate/configure deployments through versioned Workflow Service operations and restore the previous container image after a failed rollout.
How it connects
Connects Kubernetes, rolling updates, observability, deployment versioning, canary releases, and shadow mode.

Also explained in

Compare how 14 books explain rollback →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.