liveBook · Manning

Jinja

As explained in Data Pipelines with Apache Airflow, Second Edition

Jinja-based templating is the generation of text by evaluating expressions and substitutions embedded in a Jinja template.

Jinja-based templating generates dynamic text by evaluating expressions embedded in a template. In Airflow, it fills task arguments at run time with values from the task context, such as logical_date, instead of hardcoding dates or paths. section 5.2.1 section 5.5

Date-partitioned event files

A BashOperator can write each execution’s API response to a date-specific file: `curl -o /data/events/{{ logical_date | ds }}.json ...`. A downstream PythonOperator can receive matching input and output paths through `op_kwargs`, such as `/data/events/{{logical_date | ds}}.json` and `/data/stats/{{logical_date | ds}}.csv`. section 3.3.1

Practical details shape how templates behave:

  • Use double curly braces, such as `{{ name }}` or `{{ logical_date }}`, to insert runtime values. section 5.2.1
  • Format datetime components inside templates when external names require padding, for example `{{ '{:02}'.format(data_interval_start.hour) }}`. section 5.2.1
  • Check an operator’s templated-attribute allowlist; unsupported arguments preserve template text literally. section 5.2.1
  • Treat PythonOperator differently: pass run-time values through `op_args` or `op_kwargs`, rather than expecting its callable variables to be Jinja-rendered. section 5.5
Listing 3.4 Using templating to specify dates
fetch_events = BashOperator(
    task_id="fetch_events",
    bash_command=(
         "mkdir -p /data/events && "
         "curl -o /data/events/{{ logical_date | ds }}.json" #1
         "'http://events-api:8081/events/latest"
    ),
)

Shows a complete BashOperator command that renders logical_date into a daily output filename. section 3.3.1

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Defines interpolation, context values, datetime formatting, templated-attribute allowlists, and PythonOperator exceptions.
Examples
Demonstrates user-name interpolation, Wikipedia download URLs, date-partitioned event files, and dynamically generated SQL.
What you can do after
Apply it via Airflow operators and verify rendered arguments with the `airflow tasks render` command.
How it connects
Connects Jinja with Airflow context, logical_date, BashOperator, PythonOperator, SQL queries, and task rendering.

Also explained in

Compare how 4 books explain Jinja →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.