Distributed machine learning pipeline

As explained in Distributed Machine Learning Patterns

An orchestrated sequence of machine learning tasks distributed across multiple compute resources or services.

A distributed machine learning pipeline coordinates machine-learning tasks across compute resources or services to handle production scale. Careful component design can accelerate development and deployment, automate tasks, and exploit hardware acceleration. section 1.3.2

Common distributed patterns assign data, computation, or model state across workers:

• Asynchronous workers consume dataset partitions and update a centralized model, applying a work-queue pattern to speed training. section 1.3.2
• Parameter servers store and update model partitions while workers process corresponding dataset partitions, supporting models too large for one server. section 1.3.2

End-to-end image-classification pipeline

A practical system ingests the Fashion-MNIST dataset, trains and optimizes a classification model, and serves the final model for high-performance predictions. TensorFlow and Python build the model, Kubeflow runs distributed training on Kubernetes, and Argo Workflows orchestrates the pipeline. section 7.1.1 section 1.4

Figure 1.5 An example of applying the work-queue pattern in distributed machine learning systems

Shows how workers consume database-held data partitions as distributed work items while training a centralized model. section 1.3.2

How Distributed Machine Learning Patterns covers it

How deep it goes

Connects architectural patterns to an implemented end-to-end system but defers detailed component implementation to dedicated technologies and later stages.

Examples

Uses Fashion-MNIST ingestion, image-classification training, real-time prediction serving, partitioned workers, and parameter-server model storage.

What it assumes

Assumes Python machine-learning experience and some production exposure while introducing distributed-system technologies without requiring prior distributed-systems expertise.

What you can do after

Build it from scratch using TensorFlow, Python, Docker, Kubernetes, Kubeflow, and Argo Workflows across a distributed cluster.

How it connects

Connects work-queue patterns, parameter servers, distributed model training, Kubernetes, Kubeflow, Docker, and workflow orchestration.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Related concepts

• Argoproduced by · Argo is an open-source, Kubernetes-native suite of tools for continuous delivery, workflow orchestration, and workload automation.
• Dockerused by · Docker is a platform and ecosystem for building, distributing, and running applications in containers.
• TensorFlowused by · TensorFlow is an open-source software platform for numerical computation and machine learning, with tools for developing, training, and deploying models.
• fan-inused by · Fan-in is a pattern in which outputs from multiple concurrent or upstream sources converge into a single stream, task, or operation.
• fan-outused by · Fan-out is a pattern in which one input or operation distributes work, messages, or tasks across multiple downstream branches or processing units.
• machine learning workflowkind of · A machine learning workflow is an organized sequence of activities for formulating, developing, evaluating, deploying, operating, and maintaining a machine learning system.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.