Argo

As explained in Distributed Machine Learning Patterns

Argo Project is a suite of open-source tools for deploying and running applications and workloads on Kubernetes.

Argo is an open-source Kubernetes tool suite for deploying and running applications and workloads. Its core projects cover declarative GitOps delivery, progressive delivery, event-driven dependencies, and container-native workflows. section 8.4

Use the core projects for distinct operational roles:

• Argo CD manages application definitions, configurations, and environments declaratively in Git, with automated and auditable deployment lifecycle management. section 8.4
• Argo Rollouts adds Kubernetes controllers and custom resources for blue-green and canary deployment, analysis, experimentation, and progressive delivery. section 8.4
• Argo Events resolves dependencies from sources such as webhooks, S3, schedules, and streams before triggering Kubernetes objects. section 8.4
• Argo Workflows orchestrates compute-intensive container jobs as Kubernetes custom resources, using sequential steps or task dependency graphs. section 8.4

For machine learning pipelines, Argo Workflows connects independently implemented stages into an end-to-end Kubernetes workflow. Each step can run in a separate container; workflows can express parallel work with a DAG or branch conditionally from a step result. Argo Events can trigger these workflows from external events. section 8.4 section 9.4

Submitting and inspecting a workflow

Define a Workflow with an entrypoint template, container image, and command, then submit it with kubectl. Inspect workflow status, select its Pods by workflow label, and retrieve container logs. Resource templates can create Kubernetes objects such as ConfigMaps; script templates can run embedded Python. A diamond DAG makes two tasks depend on A and a final task depend on both; a coin-flip workflow conditionally runs either branch. section 8.4.1

Figure 8.9 Argo Workflows UI illustrating a map-reduce-style workflow

Inspect the workflow UI’s DAG view to see a map-reduce-style container workflow and its task relationships. section 8.4.1

An end-to-end machine learning workflow can sequence data ingestion, distributed training, and model serving components. Step memoization avoids repeating a time-consuming, redundant ingestion step in later executions. section 9.4 section 9.5

How Distributed Machine Learning Patterns covers it

How deep it goes

Explains workflow templates, submission, Pod inspection, DAG dependencies, conditional steps, and memoization; defers broader project implementation details.

Examples

Demonstrates hello-world, ConfigMap resource, Python script, diamond DAG, coin-flip conditional, and map-reduce-style workflows.

What it assumes

Assumes Python machine-learning experience and familiarity with Python and Bash scripts; introduces the relevant workflow technology basics.

What you can do after

Apply it via Kubernetes and Argo Workflows to define, submit, inspect, and optimize container-native machine learning pipelines.

How it connects

Connects Kubernetes, Kubeflow, TensorFlow, Docker, GitOps, progressive delivery, event-driven workflows, DAGs, and step memoization.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Platform Engineering on KubernetesArgo CD is a declarative continuous delivery tool that synchronizes application and infrastructure configuration from Git repositories to Kubernetes clusters.
• 

  liveBookmanning.com
  Effective Platform EngineeringArgo CD is a Kubernetes continuous-delivery tool that synchronizes applications with declarative configuration stored in version control.
• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersArgo Workflows is an open-source container-native workflow engine for orchestrating jobs and pipelines, commonly on Kubernetes.

Compare how 4 books explain Argo →

Related concepts

• Kubernetesuses · Kubernetes is an open-source platform for orchestrating and automating the deployment, scaling, and management of containerized workloads across computing clusters.
• repositoryuses · A software repository is an organized storage location or collection of project files and related artifacts, often managed with version history and change tracking.
• GitOpsimplements · GitOps is an operational approach that uses version-controlled declarative configuration as the source of truth and automation to apply or reconcile system state.
• Apache Airflowalternative to · Apache Airflow is an open-source platform for authoring, scheduling, and monitoring programmatically defined workflows, commonly represented as directed acyclic graphs.
• agent orchestrationkind of · Orchestration is the coordination and sequencing of agents, services, tasks, or components to carry out a larger workflow or objective.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.