Metadata records facts about each machine-learning workflow step, including execution state, data availability, resource usage, network performance, inputs, outputs, and model quality. These records turn opaque failures into operational evidence for retries, resource changes, worker replacement, and selective recovery. section 6.0 section 6.3.2
Use metadata to make workflow operations more informed:
- Record whether training data remains available and whether memory or CPU limits were exceeded before a step fails. section 6.3.2
- Collect runtime measurements such as memory usage, bandwidth, throughput, and latency to identify bottlenecks and abnormal workers. section 6.3.2 section 6.3.3
- Use recorded evidence to retry with more memory, restart lost prerequisites, replace slow workers, or avoid restarting every model from scratch. section 6.3.2 section 6.3.3
- Relate workflow steps to trained artifacts so model-management tools can build lineage and show contributing factors. section 6.3.3
Diagnosing a failed training step
A training step samples memory usage and training-data availability every five minutes. After 30 minutes, memory rises from 23 MB to 200 MB and the step fails. The workflow can retry with a larger memory request, allowing the trained model to continue to serving instead of repeating ingestion and training without evidence. section 6.3.2
Figure 6.16 An example workflow where the model training step failed, with the metadata collected showing an unexpected memory spike during runtime
Use the workflow diagram to trace how runtime metadata exposes the memory spike that informs recovery. section 6.3.2
