Apache Airflow

As explained in Delivering Machine Learning Projects: From design to deployment

Airflow is an open-source platform for authoring, scheduling, and monitoring programmatic workflows.

Apache Airflow is a pipeline engine for authoring, scheduling, executing, and monitoring complex programmatic data workflows. It abstracts task chaining and execution management across diverse technical stacks. section 4.3.3

Airflow is useful when AI and data projects contain many dependent processes:

• Manage imports and data preparation for exploratory data analysis. section 4.3.3
• Coordinate model training, evaluation, and production workflows. section 4.3.3
• Make processes inspectable while outsourcing bookkeeping and script management to an execution engine. section 4.3.3

Figure 4.1 An example DAG: it’s directed (D) as the information flows one way along the arrows. It’s acyclic (A) because there aren’t any cycles, and it’s a graph (G) because that’s what this sort of thing is called.

Use the DAG figure to inspect how directed, acyclic task dependencies represent a workflow. section 4.3.3

How Delivering Machine Learning Projects: From design to deployment covers it

How deep it goes

Explains pipeline abstraction, task chaining, execution management, and inspectability without detailing Airflow implementation.

Examples

Uses DAG-shaped workflows and processes for importing data, preparing features, training models, evaluating models, and deploying production flows.

What you can do after

Operate and inspect chained data processes while reducing manual bookkeeping, script-management effort, and workflow error.

How it connects

Connects pipeline orchestration with Hadoop, Oozie, data assets, exploratory data analysis, model training, and model evaluation.

liveBookmanning.com

Read Delivering Machine Learning Projects: From design to deployment on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Data Pipelines with Apache Airflow, Second EditionApache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows, especially data pipelines.
• 

  liveBookmanning.com
  Architecting an Apache Iceberg LakehouseApache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows as directed acyclic graphs.
• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersApache Airflow is an open-source platform for authoring, scheduling, and monitoring batch-oriented workflows.
• 

  liveBookmanning.com
  Financial AI in PracticeApache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows programmatically.

Compare how 5 books explain Apache Airflow →

Related concepts

• triggererhas part · The triggerer is an Apache Airflow component that monitors conditions for tasks performing asynchronous processing.
• schedulerhas part · A scheduler is a system component that selects, orders, dispatches, or assigns execution resources to tasks according to timing, dependency, placement, or other scheduling criteria.
• SQLAlchemyuses · SQLAlchemy is a Python SQL toolkit and object-relational mapping library for connecting to and working with relational databases.
• databasehas part · A database is an organized collection of persistent data managed by software for storage, retrieval, querying, and modification.
• Airflow API serverhas part · The Airflow API server is the service interface for viewing workflows, monitoring runs, and mediating database access for Airflow components.
• dag processorhas part · A DAG processor is an Airflow component that reads and parses DAG definitions and stores their serialized representations in the metastore.

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.