liveBook · Manning

Recovery

As explained in Designing AI Systems

Crash recovery is the detection and re-dispatch of work whose executing process or container has unexpectedly disappeared.

Crash recovery detects work abandoned by a vanished process or container and re-dispatches it elsewhere. In a multi-step workflow, recovery belongs to the platform rather than business logic, alongside retries, state management, and observability. A timeout or service failure can therefore trigger another attempt without requiring each workflow step to implement coordination infrastructure. section 2.1.7 section 9.2.1

A recoverable asynchronous job record supports three operational consumers:

  • The runtime server updates progress, completion results, and failure errors by job ID. section 9.2.1
  • The polling client reads status, progress, results, and errors independently of the workflow container. section 9.2.1
  • Recovery queries jobs by assigned endpoint and processing status, then uses the stored request to re-dispatch work after the assigned container disappears. section 9.2.1
  • The durable record therefore includes input_json, status, assigned_endpoint, result or error fields, timestamps, and optional checkpoint state. section 9.2.1

Recovering a disappeared workflow container

A job begins in processing with its original request body and assigned container recorded. If that container vanishes before completion, recovery finds the still-processing row for that endpoint, retains the job identity, and dispatches the saved request to another container. The polling client continues using the same job ID while the replacement runtime reports progress and eventually records completion or failure. section 9.2.1

Listing 9.9 Jobs table schema
CREATE TABLE jobs (
    job_id VARCHAR(64) PRIMARY KEY,    #A
    workflow_id VARCHAR(64) NOT NULL,
    status VARCHAR(20) NOT NULL    #B
        DEFAULT 'processing',
    progress_message TEXT,    #C
    input_json TEXT NOT NULL,    #D
    result_json TEXT,    #E
    error TEXT,    #F
    checkpoint_json TEXT,    #G
    assigned_endpoint VARCHAR(256),    #H
    created_at TIMESTAMP NOT NULL
        DEFAULT NOW(),
    updated_at TIMESTAMP NOT NULL
        DEFAULT NOW()
);

Inspect the jobs table schema to see the minimum durable state needed for lifecycle tracking and crash recovery. section 9.2.1

How Designing AI Systems covers it

How deep it goes
Defines durable job fields and access patterns for orphan detection and re-dispatch, but does not detail the recovery worker algorithm.
Examples
Uses a patient-intake multi-step workflow and an asynchronous research job whose container disappears during processing.
What it assumes
Assumes asynchronous jobs, container assignment, persistent job records, and client polling; treats workflow coordination infrastructure as managed support.
What you can do after
Operate/configure durable job storage so orphaned work can be identified and re-dispatched without losing its client-visible job identity.
How it connects
Connects workflow services with asynchronous jobs, runtime servers, polling endpoints, relational databases, retries, and observability.

Also explained in

Compare how 10 books explain recovery →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.