Workflow manager

As explained in Delivering Machine Learning Projects: From design to deployment

A workflow engine executes and manages ordered computational tasks and their dependencies.

A workflow manager executes and coordinates ordered computational tasks by representing them as a directed acyclic graph (DAG). It invokes scripts on target machines, manages dependencies between steps, and abstracts the complexity of chaining pipeline operations. section 4.3.3

A workflow manager helps teams:

• Specify a pipeline without writing a control loop for every task. section 4.3.3
• Run scripts that load, join, transform, or clean data across stores and virtual machines. section 4.3.3
• Delegate execution and scheduling across complex data resources and technical stacks. section 4.3.3

ELT pipeline

One DAG loads Table X from System A and Table Y from System B, joins them, and removes null values. Each step can contain internal steps or invoke another workflow independently. section 4.3.3

Figure 4.1 An example DAG: it’s directed (D) as the information flows one way along the arrows. It’s acyclic (A) because there aren’t any cycles, and it’s a graph (G) because that’s what this sort of thing is called.

Use the DAG figure to trace one-way task dependencies and distinguish directed, acyclic, and graph structure. section 4.3.3

How Delivering Machine Learning Projects: From design to deployment covers it

How deep it goes

Explains DAG structure, script execution, nested workflows, and abstraction of task chaining, but not scheduler internals.

Examples

Uses an ELT DAG that loads two tables, joins them, and cleans null values.

What it assumes

Assumes familiarity with scripts, data stores, virtual machines, and data-pipeline transformations.

What you can do after

Operate data pipelines by defining dependent scripts as a DAG and delegating their execution to a workflow engine.

How it connects

Connects workflow engines with schedulers, DAGs, data pipelines, ELT, Hadoop, Oozie, and Apache Airflow.

liveBookmanning.com

Read Delivering Machine Learning Projects: From design to deployment on liveBook Get the book at manning.com

Related concepts

• GitHub Actionsincludes · GitHub Actions is GitHub’s workflow automation platform for defining and running jobs in response to repository events, schedules, and other triggers.
• pipelineused by · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
• PowerShellcontrasts with · PowerShell is Microsoft's cross-platform command shell and scripting language for administering computer systems and automating tasks.
• Apache Airflowincludes · Apache Airflow is an open-source platform for authoring, scheduling, and monitoring programmatically defined workflows, commonly represented as directed acyclic graphs.

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.