An hourly data pipeline needs a different download URL for each run. Hardcoding the date would make the task useful only once. A templating language keeps the command reusable by inserting values from the Airflow task context at run time. section 5.2 section 5.2.1
In Airflow, the way to supply those values depends on the operator:
- For a templated string argument such as BashOperator’s bash_command, Jinja replaces expressions inside double braces. A URL can use {{ logical_date.year }} and format its month, day, and hour as zero-padded values. section 5.2.1
- Check whether the argument is templatable: operators specify an allowlist of attributes, and other string arguments are left literal. section 5.2.1
- A PythonOperator callable is a function, not a string to render. It can receive a named context value such as data_interval_start or capture context with **context, then construct the URL in Python. section 5.2.2
Date-partitioned output
For a daily ratings pipeline, templates_dict can supply a PythonOperator callable with an input path such as /data/python/ratings/{{data_interval_start | ds}}.json and a corresponding rankings output path. Fetching ratings before ranking them gives each run date-specific files that can be fetched again if needed. section 9.1.3
Listing 9.7 Adding the rank_movies task
def _rank_movies(templates_dict, min_ratings=2, **_):
input_path = templates_dict["input_path"]
output_path = templates_dict["output_path"]
ratings = pd.read_json(input_path) #1
ranking = rank_movies_by_rating(ratings, min_ratings=min_ratings) #2
output_dir = os.path.dirname(output_path) #3
os.makedirs(output_dir, exist_ok=True)
ranking.to_csv(output_path, index=True) #4
rank_movies = PythonOperator( #5
task_id="rank_movies",
python_callable=_rank_movies,
templates_dict={
"input_path":
"/data/python/ratings/{{data_interval_start | ds}}.json",
"output_path":
"/data/python/rankings/{{data_interval_start | ds}}.csv",
},
)
fetch_ratings >> rank_movies #6
The ranking task shows templated input and output paths in templates_dict, followed by its dependency on the ratings-fetch task. section 9.1.3
Choose a templated argument when a command or path needs runtime values; use context inside a Python callable when the value is assembled in Python. To inspect an operator’s rendered arguments before running its task, use airflow tasks render. section 5.2.2 section 5.5
