The metadata pattern records descriptive information about machine-learning workflow steps, including execution conditions, outcomes, performance signals, and produced artifacts. It turns otherwise opaque pipeline behavior into evidence for operational decisions. section 6.0 section 6.3.2
Use recorded metadata to:
- Investigate failed steps and choose responses that conserve computational resources and reduce effects on users. section 6.3.2
- Search, filter, and analyze historical pipeline artifacts, such as performant models and datasets associated with strong training metrics. section 6.3.2
- Detect network bottlenecks and replace slow workers when elastic scheduling and fault tolerance are available. section 6.3.3
- Build model lineage by relating workflow steps and identifying factors contributing to model artifacts. section 6.3.3
Diagnosing a slow worker
During parameter-server training, metadata shows that one worker has latency ten times higher than its peers. Operators can take down and restart that worker, assuming the training framework supports elastic scheduling and fault tolerance. section 6.3.3
Figure 6.19 An example parameter server-based model training where the worker on the right-hand side has extremely high latency (10 times the latency of the other workers), which slows down the entire model training process
Use the workflow view to trace how an anomalously slow worker impedes distributed model training. section 6.3.3
