Batching processes data in discrete, scheduled or incrementally scheduled groups instead of continuously. It suits workflows that collect, transform, or transfer complete time intervals of data. section 1.3.1
An Airflow batch pipeline typically relies on:
- A DAG defines task order and dependencies, making the workflow easier to understand, debug, rerun, and execute in parallel. section 1.5
- A schedule determines when each pipeline run occurs, while time intervals let the workflow bucket data into discrete periods. section 1.3.1 section 1.5
- The DAG processor parses workflow definitions, the scheduler queues eligible tasks, workers execute them, and the API server exposes monitoring. section 1.5
Daily ingestion
A daily data-ingestion workflow can run at a specified time, process the corresponding interval as one unit, and recompute earlier intervals through backfilling after code changes. section 1.3.1
Batching is a poor fit for real-time event handling or streaming workloads. Website click streams can be processed in bulk, but individual clicks should not be handled as continuously arriving events by a batch-oriented workflow. section 1.3.2
