Temporary resource failure

As explained in Distributed Machine Learning Patterns

A temporary failure is a transient failure condition for which retrying may succeed.

A temporary resource failure is a transient workflow failure for which retrying may succeed. Before retrying, identify the root cause, such as a lost network connection, preempted worker, or resource shortage. section 6.3.2

Use runtime metadata to distinguish retryable conditions from persistent problems. Check whether dependencies remain available and whether sufficient memory, CPU, and training workers exist; otherwise, restore inputs or defer work instead of repeating failure. section 6.3.2

A practical recovery flow is:

• Record step metadata, including input availability and resource usage. section 6.3.2
• Classify the failure as transient or persistent. section 6.3.2
• Retry only when dependencies and computational resources support another attempt; otherwise, restart prerequisite steps or run other tasks. section 6.3.2

Figure 6.14 The difference in the effectiveness of retrying for permanent and temporary failures

Compare effective retries for temporary failures with ineffective repetition for permanent failures. section 6.3.2

How Distributed Machine Learning Patterns covers it

How deep it goes

Explains root-cause checks, dependency validation, runtime metadata, and retry trade-offs without specifying a retry algorithm.

Examples

Uses lost connections, preempted workers, memory spikes, missing ingested data, and model-training retries as examples.

What it assumes

Assumes familiarity with workflow steps, dependencies, data persistence, and computational resource limits.

What you can do after

Diagnose failures with runtime metadata and choose whether to retry, restore prerequisites, or defer work.

How it connects

Connects machine learning workflows with model training, data ingestion, permanent failures, runtime metadata, and computational resources.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.