Cover of Distributed Machine Learning Patterns

What you will learn in Distributed Machine Learning Patterns

by Yuan Tang

Read it on liveBook Get the book at manning.com

Chapter by chapter

Chapter 1: Introduction to distributed machine learning systems

1.1 Large-scale machine learning · 1.1.1 The growing scale · 1.1.2 What can we do? · 1.2 Distributed systems · 1.2.1 What is a distributed system? · 1.2.2 The complexity and patterns · 1.3 Distributed machine learning systems · 1.3.1 What is a distributed machine learning system? · 1.3.2 Are there similar patterns? · 1.3.3 When should we use a distributed machine learning system? · 1.3.4 When should we not use a distributed machine learning system? · 1.4 What we will learn in this book · 1.5 Summary

What you learn in chapter 1 →

Chapter 2: Data ingestion patterns

2.1 What is data ingestion? · 2.2 The Fashion-MNIST dataset · 2.3 Batching pattern · 2.3.1 The problem: Performing expensive operations for Fashion MNIST dataset with limited memory · 2.3.2 The solution · 2.3.3 Discussion · 2.3.4 Exercises · 2.4 Sharding pattern: Splitting extremely large datasets among multiple machines · 2.4.1 The problem · 2.4.2 The solution · 2.4.3 Discussion · 2.4.4 Exercises · 2.5 Caching pattern · 2.5.1 The problem: Re-accessing previously used data for efficient multi-epoch model training · 2.5.2 The solution · 2.5.3 Discussion · 2.5.4 Exercises · 2.7 Section 2.3.4 · 2.8 Section 2.4.4 · 2.9 Section 2.5.4 · 2.10 Summary

What you learn in chapter 2 →

Chapter 3: Distributed training patterns

3.1 What is distributed training? · 3.2 Parameter server pattern: Tagging entities in 8 million YouTube videos · 3.2.1 The problem · 3.2.2 The solution · 3.2.3 Discussion · 3.2.4 Exercises · 3.3 Collective communication pattern · 3.3.1 The problem: Improving performance when parameter servers become a bottleneck · 3.3.2 The solution · 3.3.3 Discussion · 3.3.4 Exercises · 3.4 Elasticity and fault-tolerance pattern · 3.4.1 The problem: Handling unexpected failures when training with limited computational resources · 3.4.2 The solution · 3.4.3 Discussion · 3.4.4 Exercises · 3.6 Section 3.2.4 · 3.7 Section 3.3.4 · 3.8 Section 3.4.4 · 3.9 Summary

What you learn in chapter 3 →

Chapter 4: Model serving patterns

4.1 What is model serving? · 4.2 Replicated services pattern: Handling the growing number of serving requests · 4.2.1 The problem · 4.2.2 The solution · 4.2.3 Discussion · 4.2.4 Exercises · 4.3 Sharded services pattern · 4.3.1 The problem: Processing large model serving requests with high-resolution videos · 4.3.2 The solution · 4.3.3 Discussion · 4.3.4 Exercises · 4.4 The event-driven processing pattern · 4.4.1 The problem: Responding to model serving requests based on events · 4.4.2 The solution · 4.4.3 Discussion · 4.4.4 Exercises · 4.6 Section 4.2 · 4.7 Section 4.3 · 4.8 Section 4.4 · 4.9 Summary

What you learn in chapter 4 →

Chapter 5: Workflow patterns

5.1 What is workflow? · 5.2 Fan-in and fan-out patterns: Composing complex machine learning workflows · 5.2.1 The problem · 5.2.2 The solution · 5.2.3 Discussion · 5.2.4 Exercises · 5.3 Synchronous and asynchronous patterns: Accelerating workflows with concurrency · 5.3.1 The problem · 5.3.2 The solution · 5.3.3 Discussion · 5.3.4 Exercises · 5.4 Step memoization pattern: Skipping redundant workloads via memoized steps · 5.4.1 The problem · 5.4.2 The solution · 5.4.3 Discussion · 5.4.4 Exercises · 5.6 Section 5.2 · 5.7 Section 5.3 · 5.8 Section 5.4 · 5.9 Summary

What you learn in chapter 5 →

Chapter 6: Operation patterns

6.1 What are operations in machine learning systems? · 6.2 Scheduling patterns: Assigning resources effectively in a shared cluster · 6.2.1 The problem · 6.2.2 The solution · 6.2.3 Discussion · 6.2.4 Exercises · 6.3 Metadata pattern: Handle failures appropriately to minimize the negative effect on users · 6.3.1 The problem · 6.3.2 The solution · 6.3.3 Discussion · 6.3.4 Exercises · 6.5 Section 6.2 · 6.6 Section 6.3 · 6.7 Summary

What you learn in chapter 6 →

Chapter 7: Project overview and system architecture

7.1 Project overview · 7.1.1 Project background · 7.1.2 System components · 7.2 Data ingestion · 7.2.1 The problem · 7.2.2 The solution · 7.2.3 Exercises · 7.3 Model training · 7.3.1 The problem · 7.3.2 The solution · 7.3.3 Exercises · 7.4 Model serving · 7.4.1 The problem · 7.4.2 The solution · 7.4.3 Exercises · 7.5 End-to-end workflow · 7.5.1 The problems · 7.5.2 The solutions · 7.5.3 Exercises · 7.7 Section 7.2 · 7.8 Section 7.3 · 7.9 Section 7.4 · 7.10 Section 7.5 · 7.11 Summary

What you learn in chapter 7 →

Chapter 8: Overview of relevant technologies

8.1 TensorFlow: The machine learning framework · 8.1.1 The basics · 8.1.2 Exercises · 8.2 Kubernetes: The distributed container orchestration system · 8.2.1 The basics · 8.2.2 Exercises · 8.3 Kubeflow: Machine learning workloads on Kubernetes · 8.3.1 The basics · 8.3.2 Exercises · 8.4 Argo Workflows: Container-native workflow engine · 8.4.1 The basics · 8.4.2 Exercises · 8.6 Section 8.1 · 8.7 Section 8.2 · 8.8 Section 8.3 · 8.9 Section 8.4 · 8.10 Summary

What you learn in chapter 8 →

Chapter 9: A complete implementation

9.1 Data ingestion · 9.1.1 Single-node data pipeline · 9.1.2 Distributed data pipeline · 9.2 Model training · 9.2.1 Model definition and single-node training · 9.2.2 Distributed model training · 9.2.3 Model selection · 9.3 Model serving · 9.3.1 Single-server model inference · 9.3.2 Replicated model servers · 9.4 The end-to-end workflow · 9.4.1 Sequential steps · 9.4.2 Step memoization · 9.5 Summary

What you learn in chapter 9 →

Concepts explained in this book

Model ServingA software system that receives prediction requests and returns outputs from a deployed machine learning model.
machine learning workflowThe stages and activities involved in developing, deploying, operating, maintaining, and retiring machine learning models and systems.
data ingestionData ingestion is the process of acquiring, importing, and preparing data for downstream computation or analysis.
distributed trainingThe process of fitting a machine learning model by coordinating computation across multiple machines or execution units.
parameter server patternA parameter server is a server component that stores and updates a designated portion of a machine learning model's parameters.
replicated service patternA model-serving deployment pattern consisting of multiple server instances that host equivalent model replicas.
collective communicationA collective communication pattern coordinates communication of computation updates among multiple workers and synchronizes their model copies.
allreduceAn allreduce is a collective communication operation that aggregates values across participating processes and distributes the aggregate to every process.
gang schedulingGang scheduling starts or schedules a group of cooperating workers together so they are simultaneously available for coordinated computation or communication.
step memoizationA workflow optimization pattern that caches a step's results or metadata so redundant executions can be skipped when relevant inputs and conditions are unchanged.
fair-share schedulingFair-share scheduling is an operating-system scheduling algorithm that distributes CPU usage among users or groups rather than equally among individual processes.
workerA machine or execution unit that processes assigned data and participates in distributed model training.
priority-based schedulingPriority scheduling is a scheduling algorithm that orders work according to assigned priority levels and may preempt lower-priority work.
sequential processingA workflow pattern in which dependent processing proceeds in coordinated order, with an operation waiting for required preceding work to complete.
event-driven architectureA processing architecture in which computation is initiated in response to incoming events or requests.
distributed learning systemA distributed machine learning system is a machine learning system whose computation, data processing, or model operations are distributed across multiple machines or execution nodes.
convolutional neural networkA convolutional neural network is a neural network model that uses convolutional operations to learn spatial or locally structured representations.
ArgoArgo Project is a suite of open-source tools for deploying and running applications and workloads on Kubernetes.
model selectionThe process of choosing a machine learning model or configuration from a set of candidates according to an evaluation criterion.
preprocessingThe transformation and preparation of raw data into a form suitable for downstream computation or model training.
TensorFlowAn open-source platform for numerical computation and machine learning, including tools for building and training neural networks.
hyperparameterA configuration value chosen before or during model training that controls the model structure or training procedure rather than being learned directly from the data.
fan-inA workflow pattern that combines results from multiple upstream processing branches before downstream processing.
fan-outA fan-out pattern starts multiple separate workflow steps from a single upstream step so they can process the same input or output.
KubernetesKubernetes is an open-source platform for automating the deployment, scaling, and management of containerized applications.
model partitionA subset of a machine learning model's parameters assigned to a particular storage or computation component.
batching patternA data-processing pattern that divides a dataset into small batches so processing can operate within available memory.
epochAn epoch is one complete pass through every training example in a dataset during model training.
persistent volumeA PersistentVolume is a Kubernetes storage resource representing durable storage that can be provisioned and mounted independently of an individual workload.
sharded service patternA service architecture pattern that divides a large request or workload into sub-requests processed by multiple shards and combines their results.
parameterA model parameter is a learned numerical value in a machine learning model that determines the model's behavior.
metadataMetadata is data that describes the execution state, resource usage, inputs, outputs, or other properties of a workflow step.
metadata patternA workflow pattern that records and uses descriptive information about processing steps, execution conditions, and their outcomes.
persistent volume claimA PersistentVolumeClaim is a Kubernetes resource that requests persistent storage with specified capacity and access requirements.
machine learning systemA software system that collects data, trains or serves machine learning models, and supports the surrounding operational workflow.
horizontal scalingA scaling technique that increases capacity by adding equivalent service instances or machines rather than enlarging one instance.
deadlockA deadlock is a state in which a set of tasks cannot proceed because each is waiting for resources or progress that cannot become available.
process groupA worker process group is a set of worker processes that participate together in distributed computation and collective communication.
NumPy arrayA NumPy array is a multidimensional, in-memory data structure provided by the NumPy library.
directed acyclic graphA directed acyclic graph is a directed graph with no path that returns to an earlier vertex, so its dependencies have no cycles.
DockerfileA Dockerfile is a declarative text file containing instructions for building a Docker container image.
TFJobA TFJob spec is the configuration specification for a Kubeflow TensorFlow training job, including distributed job components and replica counts.
failureAn event in which a component, operation, or workflow step does not complete as intended.
broadcast operationA collective communication operation that sends one value or result to every process in a participating process group.
InferenceServiceAn InferenceService is a KServe resource specification describing a deployed machine learning inference service and its model source and serving configuration.
stale gradientA gradient computed using a model version older than the latest version available for updating the model.
time-based cacheA cache whose validity or freshness is determined by the elapsed time since the stored data was last updated.
Kubeflow training operatorThe Kubeflow Training Operator is a Kubernetes operator that manages distributed and nondistributed machine learning training jobs through custom resources.
LeNetLeNet is a convolutional neural network architecture originally designed for handwritten-digit recognition.
feature engineeringFeature engineering is the process of creating, transforming, or selecting input features to improve a machine learning model.
denial-of-service attackA denial-of-service attack is an intentional or accidental activity that overwhelms a service with requests and prevents authorized access or normal operation.
rate limitRate limiting is a technique that restricts the number or frequency of requests a service processes within a specified interval or concurrency bound.
datasetA collection of data records or examples used for machine learning or other computational analysis.
memory errorAn error condition that occurs when a process cannot obtain enough memory to continue execution.
distributed machine learning pipelineAn orchestrated sequence of machine learning tasks distributed across multiple compute resources or services.
HyperbandA hyperparameter-optimization algorithm that allocates computational resources adaptively among configurations using successive halving.
hyperparameter tuningHyperparameter tuning is the process of selecting hyperparameter values by evaluating models trained with different configurations.
KerasKeras is a high-level deep-learning API for defining, training, evaluating, and exporting neural-network models, commonly used through TensorFlow as tf.keras.
KerasTunerKeras Tuner is a library for searching and selecting hyperparameter configurations for Keras models.
optimizerAn optimizer is an algorithm that updates model parameters to minimize a training objective such as loss.
distributed model serving systemThe process of serving a machine learning model across multiple networked machines, typically with large datasets or models partitioned or stored in distributed infrastructure.
convolutional layerA convolutional layer is a neural-network layer that applies learned convolutional filters to its input.
kubectlkubectl is the command-line client for interacting with Kubernetes clusters and their API resources.
custom objectA Kubernetes custom resource is an API object of a user-defined resource type that extends the Kubernetes API.
dropoutA neural-network layer that randomly omits units during training to reduce overfitting and model complexity.
KubeflowAn open-source platform for deploying and orchestrating portable, scalable machine learning workflows on Kubernetes.
distribution strategyA distribution strategy is a mechanism that specifies how computation and model variables are replicated and coordinated across devices or workers.
SavedModelTensorFlow SavedModel is TensorFlow's serialized model format for storing a model and its serving information for later consumption.
TF_CONFIGTF_CONFIG is a TensorFlow environment-variable configuration containing cluster and task information for distributed execution.
tf.distributetf.distribute is TensorFlow's API module for distributing computation and model training across devices and workers.
permanent failureA permanent failure is a failure condition expected to persist such that retrying is ineffective.
retryA retry is the repeated execution of a failed workflow step.
temporary resource failureA temporary failure is a transient failure condition for which retrying may succeed.
training completionTraining completion is the proportion of the planned model-training work that has been completed.
content-based cacheA cache whose stored result is identified and retrieved according to the content or configuration associated with the computation.
model server shardA serving component that processes a subset of a large inference request as part of a sharded model-serving deployment.
podA pod is the smallest deployable unit in Kubernetes, consisting of one or more co-located containers sharing specified resources and execution context.
gradientA gradient is a vector of partial derivatives that indicates how a function changes with respect to its parameters.
computer clusterA cluster is a group of connected computing machines managed to execute workloads as a coordinated resource.
batch normalizationBatch normalization is a neural-network technique that normalizes intermediate activations using batch statistics during training.
KServeKServe is a Kubernetes-based model serving platform that provides standardized resources and abstractions for deploying machine learning models.
sharding functionA deterministic function that maps a request or sub-request to the shard responsible for processing it.
conditional syntaxConditional syntax is a workflow-language feature that controls whether a step executes based on a condition.
resource templateA resource template is an Argo Workflows template that submits a Kubernetes resource as part of workflow execution.
TensorFlow I/OA TensorFlow extension library that provides additional filesystem and file-format integrations for data input.
when clauseA when clause is an Argo Workflows expression that specifies a condition for executing a workflow step.
k3dA lightweight command-line wrapper for running k3s clusters in Docker.
K3sA lightweight Kubernetes distribution designed for resource-constrained or streamlined environments.
Kubernetes namespaceA Kubernetes resource that provides an isolation boundary for groups of resources within a cluster.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.