4 Templating tasks using the Airflow context

 

This chapter covers

  • Rendering variables at runtime with templating
  • Master variable templating with the PythonOperator and across all other operators
  • Rendering templated variables for debugging purposes
  • Performing operations on external systems

In the previous chapters, we touched the surface of how DAGs and operators work together and how to schedule a workflow in Airflow. In this chapter, we look in-depth at what operators represent, what they are, how they function, and when and how they are executed. We also demonstrate how operators can be used to communicate with remote systems via hooks, which allows you to perform tasks such as loading data into a database, running a command in a remote environment, and performing workloads outside of Airflow.

4.1 Inspecting data for processing with Airflow

4.1.1 Determining how to load incremental data

4.2 Task context and Jinja templating

4.2.1 Templating operator arguments

4.2.2 Templating the PythonOperator

4.2.3 Passing additional variables to the PythonOperator

4.2.4 Inspecting templated arguments

4.2.5 What is available for templating?

4.3 Bringing it all together

4.4 Summary