Training completion measures the proportion of planned model-training work finished. Reaching 100% means all requested batches and epochs have been processed, but completion should be read alongside model accuracy. section 6.3.2 section 6.3.2
Use completion metadata to make operational decisions:
- Compare progress across model-training steps and detect work advancing too slowly. section 6.3.2
- Prioritize computational resources for promising steps instead of wasting time on runs likely to produce low-accuracy models. section 6.3.2
- Combine completion with accuracy and failure metadata when deciding whether to continue, retry, or disregard a step. section 6.3.2
Slow progress under limited time
A training step advancing by only 1% every 30 minutes may be deprioritized when its limited remaining time makes a low-accuracy model likely. section 6.3.2
Figure 6.18 An example workflow where two model training steps fail. One is disregarded because it is progressing very slowly, and the model will likely have low accuracy given the limited time.
Shows a workflow in which a slowly progressing training step is disregarded while resources shift to more promising steps. section 6.3.2
