Distributed training workers run in Kubernetes Pods whose files disappear when the Pods complete. A PersistentVolumeClaim preserves checkpoints and trained models beyond that Pod lifecycle. section 9.2.2
A PersistentVolumeClaim (PVC) requests storage from a PersistentVolume (PV), much as a Pod requests CPU and memory from a node. The claim specifies capacity and access mode, while the PV persists independently of individual Pods. section 9.2.2
Use the claim to connect persistent storage to machine-learning workloads:
- Request a capacity such as 1 Gi and an access mode such as ReadWriteOnce, ReadOnlyMany, or ReadWriteMany. section 9.2.2
- Reuse the mounted storage across workflow steps to retain downloaded datasets and pass trained models to selection or serving. section 9.4.1
Distributed training storage
A two-worker TFJob mounts the claim at /trained_model. Workers write checkpoints and the saved model there; later workflow steps mount the same persistent storage for model selection and serving. section 9.4.1
