Apache Airflow
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring programmatically defined workflows, commonly represented as directed acyclic graphs.
Data Pipelines with Apache Airflow, Second Edition
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows, especially data pipelines.
- How it is used
- Orchestrates scheduled or event-triggered data pipelines whose dependent tasks may work across external systems.
- How deep it goes
- Works through DAG execution, local setup, failure recovery, and production-operation choices rather than stopping at a tool definition.
- Examples
- Uses a local standalone installation, Docker Compose services, UI views, and a failed download task.
- What you can do after
- Build it from scratch: define and run a DAG, then diagnose failed tasks and selectively rerun them through the UI.
Read how Data Pipelines with Apache Airflow, Second Edition explains Apache Airflow →
Architecting an Apache Iceberg Lakehouse
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows as directed acyclic graphs.
- How it is used
- Orchestrates conditional, retryable, and observable Apache Iceberg maintenance workflows at lakehouse scale.
- How deep it goes
- Explains DAG structure and demonstrates metadata checks, task dependencies, retries, and Spark-triggered compaction, while deferring broader platform internals.
- Examples
- Uses a daily compaction DAG, conditional small-file checks, snapshot expiration, orphan-file removal, and Prometheus metric emission.
- What you can do after
- Operate/configure Iceberg maintenance by defining Airflow DAGs that inspect metadata and trigger Spark SQL procedures conditionally.
Read how Architecting an Apache Iceberg Lakehouse explains Apache Airflow →
Delivering Machine Learning Projects: From design to deployment
Airflow is an open-source platform for authoring, scheduling, and monitoring programmatic workflows.
- How it is used
- Orchestrates data pipelines supporting import, preparation, exploratory analysis, model training, evaluation, and production.
- How deep it goes
- Explains pipeline abstraction, task chaining, execution management, and inspectability without detailing Airflow implementation.
- Examples
- Uses DAG-shaped workflows and processes for importing data, preparing features, training models, evaluating models, and deploying production flows.
- What you can do after
- Operate and inspect chained data processes while reducing manual bookkeeping, script-management effort, and workflow error.
Read how Delivering Machine Learning Projects: From design to deployment explains Apache Airflow →
Designing Deep Learning Systems: A guide for software engineers
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring batch-oriented workflows.
- How it is used
- Orchestrates scheduled, batch-oriented ETL and data workflows through code, task dependencies, execution management, and monitoring.
- How deep it goes
- Explains DAGs, operators, dependencies, dynamic tasks, interfaces, and a representative workflow, but does not detail platform internals.
- Examples
- Demonstrates a daily pipeline that senses new data, transforms it, creates a database table, and exposes execution through UI and CLI commands.
- What you can do after
- Operate scheduled workflows by defining Python DAGs, assigning operators, declaring dependencies, deploying code, and inspecting execution status.
Read how Designing Deep Learning Systems: A guide for software engineers explains Apache Airflow →
Financial AI in Practice
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows programmatically.
- How it is used
- Orchestrates scheduled data tasks for credit-data ingestion and monthly merges.
- How deep it goes
- Defines a minimal monthly DAG and identifies production controls, but stops short of a full production pipeline.
- Examples
- Merges monthly usage, bureau, and demographic data into a credit data mart and mentions a Kaggle credit-data example.
- What you can do after
- Apply Airflow to schedule and orchestrate data tasks in a credit-scoring workflow while accounting for production controls.
Also covered in
- Learn AI Data EngineeringIn early access (MEAP)
- Effective Data Analysis
What to know first
- PythonPython is a high-level, general-purpose, dynamically typed programming language with readable syntax and support for multiple programming paradigms.
- pipelineA pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.
- agent orchestrationAgent orchestration is the coordination of agents, their tasks, tools, and interactions to accomplish a larger objective or workflow.
- directed acyclic graphA directed acyclic graph is a graph whose edges have directions and contain no directed cycles.
Related concepts
- Airflow API serverhas part · The Airflow API server is the service interface for viewing workflows, monitoring runs, and mediating database access for Airflow components.
- dag processorhas part · A DAG processor is an Airflow component that reads and parses DAG definitions and stores their serialized representations in the metastore.
- schedulerhas part · A scheduler is a software component that selects, orders, or dispatches tasks for execution and may assign them execution resources according to defined policies or constraints.
- databasehas part · A database is an organized persistent data store, typically managed by software, that supports storing, retrieving, and modifying data.
- SQLAlchemyuses · SQLAlchemy is a Python SQL toolkit and object-relational mapping library for connecting to and working with relational databases.
- triggererhas part · The triggerer is an Apache Airflow component that monitors conditions for tasks performing asynchronous processing.
Questions about Apache Airflow
What is Apache Airflow?
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring programmatically defined workflows, commonly represented as directed acyclic graphs.
Which Manning books explain Apache Airflow?
Data Pipelines with Apache Airflow, Second Edition, Architecting an Apache Iceberg Lakehouse, Delivering Machine Learning Projects: From design to deployment, Designing Deep Learning Systems: A guide for software engineers and Financial AI in Practice.
What should I know before learning Apache Airflow?
Manning books that teach Apache Airflow usually introduce Python, pipeline, agent orchestration and directed acyclic graph first.
What are the parts of Apache Airflow?
Apache Airflow includes Airflow API server, dag processor, scheduler, database and triggerer.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




