Workflow construction

As explained in Designing Deep Learning Systems: A guide for software engineers

Workflow construction is the process of decomposing code into computational steps and specifying their ordering and dependencies for orchestrated execution.

Workflow construction converts locally tested prototype code into a production workflow, but repeated transitions expose a costly gap between experimentation and orchestration. section 9.1.3

Treat workflow construction as an iterative engineering activity:

• Learn the orchestration system’s workflow DAG syntax, libraries, and coding paradigms. section 9.1.3
• Debug execution across remote workflow environments and logs when local execution is unavailable. section 9.1.3
• Update the workflow repeatedly as iterative deep learning development tests new improvements from local prototypes to production experiments. section 9.1.3

Figure 9.3 A data scientist’s view of deep learning project development

Use the development-flow diagram to trace how prototyping and production experimentation repeatedly require workflow changes. section 9.1.3

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Details DAG syntax, libraries, coding paradigms, remote debugging, and repeated updates, but does not provide construction procedures or code.

Examples

Uses a local-to-production development flow and remote workflow execution logs to illustrate recurring construction and debugging.

What it assumes

Assumes locally tested algorithm prototypes and introduces the orchestration-specific burden of turning them into production workflows.

How it connects

Connects workflow orchestration with deep learning, prototyping, production experimentation, workflow DAGs, and remote execution.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.