Distributed training workers run in Kubernetes Pods, so their local files disappear when the Pods complete. A PersistentVolume (PV) supplies cluster storage whose lifecycle is independent of those Pods, preserving checkpoints and trained models for later serving. section 9.2.2
A PV may be provisioned by an administrator or dynamically provisioned. A PersistentVolumeClaim (PVC) is the user’s storage request: it specifies capacity and access mode, while Pods consume the resulting storage through a mounted volume. section 9.2.2
A typical machine-learning workflow uses the volume to carry artifacts between isolated execution steps:
- Reuse the same claim across workflow steps so ingestion can persist a dataset and training can persist the model for selection and serving. section 9.4.1
Distributed training with durable output
Two TFJob worker replicas mount the training claim at /trained_model. The workers write saved models and checkpoints to that path; after the worker Pods finish, those artifacts remain available to downstream model-selection and serving steps. section 9.4.1
