tf.distribute extends local TensorFlow training across multiple workers. MultiWorkerMirroredStrategy synchronously replicates model variables on every device and coordinates updates through collective communication such as all-reduce. section 9.2.2
A distributed training implementation must:
- Create the strategy with automatic collective communication so available hardware can be selected. section 9.2.2
- Batch and shard the input pipeline, then create the model inside strategy.scope() so variables replicate correctly. section 9.2.2
- Train with fit() outside the scope and save the completed model only from the chief worker. section 9.2.2
Two-worker deployment
A containerized TensorFlow script can run as a Kubeflow TFJob with two worker replicas, shared persistent storage, checkpoints, and a CNN model argument.
