How deeply does Distributed Machine Learning Patterns cover distributed training?

Explains coordination patterns, bottlenecks, failures, and a TensorFlow multi-worker implementation through deployment and workflow execution.

Distributed training

As explained in Distributed Machine Learning Patterns

The process of fitting a machine learning model by coordinating computation across multiple machines or execution units.

Distributed training fits a machine learning model across multiple nodes after data ingestion. Workers concurrently consume processed data and calculate updates, allowing training to use clustered compute when datasets, models, or runtime demands exceed a single machine. section 3.1 section 3.0

Table 3.1 Comparison of traditional (nondistributed) training and distributed training for machine learning models (view table figure)

	Traditional model training	Distributed model training
Computational resources	Laptop or single remote server	Cluster of machines
Dataset location	Local disk on a single laptop or machine	Remote distributed database or partitions on disks of multiple machines
Network infrastructure	Local hosts	InfiniBand or RDMA
Model size	Small enough to fit on a single machine	Medium to large

Compare single-machine and distributed training by compute location, dataset placement, network infrastructure, and model size. section 3.1

Choose the coordination pattern by model size and communication cost:

Use parameter servers for large, complex models that cannot fit on one machine; partition model parameters across servers while workers send gradients for updates. section 3.9 section 7.3.2
Use collective communication for smaller models when parameter-server traffic becomes a bottleneck. Each worker holds a full parameter copy, computes gradients on a data partition, and synchronizes updates through collective operations such as all-reduce. section 3.9 section 7.3.2
Plan for failures as part of training: skip unusable corrupted batches, checkpoint model parameters during network instability, and resume after connectivity returns. Synchronous collective communication can block the whole job when the network is slow. section 3.4.2

Multi-worker TensorFlow job

For synchronous multi-worker training, create a tf.distribute.MultiWorkerMirroredStrategy, define the distributed input pipeline and model inside its strategy scope, then call fit(). The strategy copies layer variables across workers and uses collective communication. Save the completed model only from the chief worker, identified through TF_CONFIG, to avoid incorrect or redundant output. section 9.2.2

Operational deployment can package the training script in a container and run a two-worker Kubeflow TFJob on Kubernetes with persistent storage for checkpoints and saved models. An Argo workflow can place data ingestion before parallel distributed runs for CNN, dropout, and batch-normalization variants, then select a top model and create model serving. section 9.2.2 section 9.4.1

How Distributed Machine Learning Patterns covers it

Cover of Distributed Machine Learning Patterns

liveBook manning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Designing Deep Learning Systems: A guide for software engineersDistributed training is the process of training a machine learning model across multiple computing devices or processes, often with data partitioned among workers.
liveBook manning.com

Deep Learning with Python, Third EditionA training approach that spreads computation and/or data across multiple devices to speed up model training.
liveBook manning.com

Deep Learning with R, Third EditionDistributed training is the execution of model training across multiple processors or machines that coordinate computation and parameter updates.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA training approach that distributes computation or data across devices or machines.
liveBook manning.com

Graph Neural Networks in ActionDistributed training is a technique in which model computation and data processing are divided across multiple computing processes or devices.

Compare how 6 books explain distributed training →

Related concepts

data parallelincludes · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
model parallelismincludes · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
pipeline parallelismincludes · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
TF_CONFIGconfigured by · TF_CONFIG is an environment-variable configuration that specifies cluster and task information for TensorFlow distributed execution.
distribution strategykind of · A distribution strategy is a mechanism that specifies how computation and model variables are replicated and coordinated across devices or workers.
JAXused by · JAX is a Python library for high-performance array-based numerical computing with composable transformations including automatic differentiation, compilation, vectorization, and parallelization.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.