4 Templating Tasks Using the Airflow Context


This chapter covers

  • Rendering variables at runtime with templating
  • Variable templating with the PythonOperator vs other operators
  • Rendering templated variables for debugging purposes
  • Performing operations on external systems

In the previous chapters, we touched the surface of how DAGs and operators work together and how scheduling a workflow works in Airflow. In this chapter, we have in-depth coverage of what operators represent, what they are, how they function, and when and how they are executed. Besides these concepts, we demonstrate how operators can be used to communicate with remote systems via hooks, which allows you to perform tasks such as loading data into a database, running a command in a remote environment, and performing workloads somewhere else than in Airflow.

4.1       Inspecting data for processing with Airflow

4.1.1   Determining how to load incremental data

4.2       Task context & Jinja templating

4.2.1   Templating operator arguments

4.2.2   What is available for templating?

4.2.3   Templating the PythonOperator

4.2.4   Providing variables to the PythonOperator

4.2.5   Inspecting templated arguments

4.3       Hooking up other systems

4.4       Summary