concept performance in category apache airflow
This is an excerpt from Manning's book Data Pipelines with Apache Airflow MEAP V05.
In practice, this means carefully thinking about your data sources and determining if all these data sources are really required. For the datasets that are needed, you can try to see if you can reduce the size of the required datasets by discarding rows/columns that aren’t used. Performing aggregations early on can also substantially increase performance, as the right aggregation can greatly reduce the size of an intermediate dataset - thus decreasing the amount of work that needs to be done downstream.
As explained in Section 12.1, the executor is part of Airflow’s scheduler. The DAG processor and task scheduler can only be run in one single way, by starting airflow scheduler. However, the task executor can be installed in different ways, from a single process on a single machine to multiple processes on multiple machines, for performance and/or redundancy.
