Apache Airflow
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring programmatically defined workflows, commonly represented as directed acyclic graphs.
Data Pipelines with Apache Airflow, Second Edition
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows, especially data pipelines.
- How it is used
- Orchestrates scheduled or event-triggered data pipelines whose dependent tasks may work across external systems.
- How deep it goes
- Works through DAG execution, local setup, failure recovery, and production-operation choices rather than stopping at a tool definition.
- Examples
- Uses a local standalone installation, Docker Compose services, UI views, and a failed download task.
- What you can do after
- Build it from scratch: define and run a DAG, then diagnose failed tasks and selectively rerun them through the UI.
Read how Data Pipelines with Apache Airflow, Second Edition explains Apache Airflow →
Architecting an Apache Iceberg Lakehouse
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows as directed acyclic graphs.
- How it is used
- Orchestrates conditional, retryable, and observable Apache Iceberg maintenance workflows at lakehouse scale.
- How deep it goes
- Explains DAG structure and demonstrates metadata checks, task dependencies, retries, and Spark-triggered compaction, while deferring broader platform internals.
- Examples
- Uses a daily compaction DAG, conditional small-file checks, snapshot expiration, orphan-file removal, and Prometheus metric emission.
- What you can do after
- Operate/configure Iceberg maintenance by defining Airflow DAGs that inspect metadata and trigger Spark SQL procedures conditionally.
Read how Architecting an Apache Iceberg Lakehouse explains Apache Airflow →
Delivering Machine Learning Projects: From design to deployment
Airflow is an open-source platform for authoring, scheduling, and monitoring programmatic workflows.
- How it is used
- Orchestrates data pipelines supporting import, preparation, exploratory analysis, model training, evaluation, and production.
- How deep it goes
- Explains pipeline abstraction, task chaining, execution management, and inspectability without detailing Airflow implementation.
- Examples
- Uses DAG-shaped workflows and processes for importing data, preparing features, training models, evaluating models, and deploying production flows.
- What you can do after
- Operate and inspect chained data processes while reducing manual bookkeeping, script-management effort, and workflow error.
Read how Delivering Machine Learning Projects: From design to deployment explains Apache Airflow →
Designing Deep Learning Systems: A guide for software engineers
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring batch-oriented workflows.
- How it is used
- Orchestrates scheduled, batch-oriented ETL and data workflows through code, task dependencies, execution management, and monitoring.
- How deep it goes
- Explains DAGs, operators, dependencies, dynamic tasks, interfaces, and a representative workflow, but does not detail platform internals.
- Examples
- Demonstrates a daily pipeline that senses new data, transforms it, creates a database table, and exposes execution through UI and CLI commands.
- What you can do after
- Operate scheduled workflows by defining Python DAGs, assigning operators, declaring dependencies, deploying code, and inspecting execution status.
Read how Designing Deep Learning Systems: A guide for software engineers explains Apache Airflow →
Financial AI in Practice
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows programmatically.
- How it is used
- Orchestrates scheduled data tasks for credit-data ingestion and monthly merges.
- How deep it goes
- Defines a minimal monthly DAG and identifies production controls, but stops short of a full production pipeline.
- Examples
- Merges monthly usage, bureau, and demographic data into a credit data mart and mentions a Kaggle credit-data example.
- What you can do after
- Apply Airflow to schedule and orchestrate data tasks in a credit-scoring workflow while accounting for production controls.
Related concepts
- triggererhas part · The triggerer is an Apache Airflow component that monitors conditions for tasks performing asynchronous processing.
- schedulerhas part · A scheduler is a system component that selects, orders, dispatches, or assigns execution resources to tasks according to timing, dependency, placement, or other scheduling criteria.
- SQLAlchemyuses · SQLAlchemy is a Python SQL toolkit and object-relational mapping library for connecting to and working with relational databases.
- databasehas part · A database is an organized collection of persistent data managed by software for storage, retrieval, querying, and modification.
- Airflow API serverhas part · The Airflow API server is the service interface for viewing workflows, monitoring runs, and mediating database access for Airflow components.
- dag processorhas part · A DAG processor is an Airflow component that reads and parses DAG definitions and stores their serialized representations in the metastore.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.