liveBook · Manning

Distribution strategy

As explained in Distributed Machine Learning Patterns

A distribution strategy is a mechanism that specifies how computation and model variables are replicated and coordinated across devices or workers.

A distribution strategy coordinates synchronous model training across workers by replicating every model variable on each device and combining worker updates through collective communication such as all-reduce. section 9.2.2

Implement the strategy around these operational decisions:

  • Create a MultiWorkerMirroredStrategy and define the input pipeline and model inside strategy.scope() so TensorFlow distributes variables correctly. section 9.2.2
  • Save the completed model only from the chief worker, using TF_CONFIG to distinguish worker identity and avoid conflicting outputs. section 9.2.2

Two-worker TensorFlow training job

Package the Python 3.9 training script with TensorFlow 2.11.0, import the image into a k3d cluster, mount persistent storage, and submit a Kubeflow TFJob with two worker replicas.

How Distributed Machine Learning Patterns covers it

How deep it goes
Implements strategy scope, batch scaling, data sharding, chief-only saving, containerization, and TFJob deployment.
Examples
Demonstrates a two-worker CNN job packaged in Docker and submitted as a Kubeflow TFJob.
What it assumes
Assumes a locally trainable model and input pipeline, then extends them into coordinated multi-worker execution.
What you can do after
Operate TensorFlow multi-worker training with automatic sharding, chief-only export, containers, and Kubeflow TFJobs.
How it connects
Connects synchronous training with collective communication, TensorFlow data pipelines, Kubernetes, Kubeflow, and persistent storage.
Cover of Distributed Machine Learning Patterns
liveBookmanning.com
Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.