Suppose a DAG downloads user events, then needs to count events by date and user. A BashOperator can fetch the file; a PythonOperator can load it with pandas, calculate the counts, and write the result. PythonOperator runs a Python callable as an Airflow task. Give the task a task_id and pass the callable—not its return value—to python_callable. The function runs when the task executes. section 3.1 section 2.2.2
Configure a callable without duplicating it
For a page-view download function that accepts output_path, configure the destination on the task: get_data = PythonOperator( task_id="get_data", python_callable=_get_data, op_kwargs={"output_path": "/tmp/wikipageviews.gz"}, ) When the task runs, output_path is passed as a keyword argument. Use op_args instead when the function expects positional arguments. section 5.2.3
Choose how the callable receives run-time values:
- Declare a named parameter such as data_interval_start to receive that task-context value, or use **context to capture context variables in a dictionary. section 5.2.2
- Put Jinja expressions in supported argument values such as op_kwargs, not inside the callable's Python code. A templated date or file path is rendered before it reaches the function. section 5.2.2 section 5.2.3
- Use templates_dict for templated paths shared with a callable. For example, a daily ratings workflow writes a date-partitioned JSON file, then a second PythonOperator reads it and writes rankings. section 9.1.3
Listing 5.11 Providing templated strings to the callable
def _get_data(year, month, day, hour, output_path, **_):
url = (
"https://dumps.wikimedia.org/other/pageviews/"
f"{year}/{year}-{month:0>2}/"
f"pageviews-{year}{month:0>2}{day:0>2}-{hour:0>2}0000.gz"
)
request.urlretrieve(url, output_path)
get_data = PythonOperator(
task_id="get_data",
python_callable=_get_data,
op_kwargs={
"year": "{{ logical_date.year }}", #1
"month": "{{ logical_date.month }}",
"day": "{{ logical_date.day }}",
"hour": "{{ logical_date.hour }}",
"output_path": "/tmp/wikipageviews-
➥ {{ logical_date.format('YYYYMMDDHH') }}.gz",
},
)
See how templated logical-date fields become arguments to a page-view download function. section 5.2.3
Keep orchestration visible where it matters. Conditional branches inside a PythonOperator can hide which path a DAG run took; separate DAG branches make that choice easier to see. A specialized operator may also handle external-system execution and connectivity with less custom code. section 6.2.1
