python_callable is the PythonOperator argument that selects the Python callable executed when a task runs. A callable is commonly a function, and any object implementing __call__() also qualifies. section 2.2.2 section 5.2.2
Use the argument to connect a task definition with execution logic:
- Define a Python function, such as _get_pictures or _get_data. section 2.2.2
- Pass that function to PythonOperator through python_callable. section 2.2.2
- Let the operator call the function when the task executes. section 2.2.2
- Accept selected Airflow context variables as named function arguments, or capture all supplied context with **context. section 5.2.2
Using task context to build a download URL
def _get_data(data_interval_start, **context): year, month, day, hour, *_ = data_interval_start.timetuple() # Build a URL from the interval components get_data = PythonOperator( task_id="get_data", python_callable=_get_data, ) The function receives data_interval_start directly while additional context can be captured through **context. section 5.2.2
Listing 2.6 Running a Python function using the PythonOperator
def _get_pictures(): #1 # Ensure directory exists pathlib.Path("/tmp/images").mkdir(parents=True, exist_ok=True) #2 # Download all pictures in launches.json with open("/tmp/launches.json") as f: #3 launches = json.load(f) image_urls = [l["image"] for l in launches["results"]] for image_url in image_urls: try: response = requests.get(image_url) #4 image_filename = image_url.split("/")[-1] target_file = f"/tmp/images/{image_filename}" with open(target_file, "wb") as f: f.write(response.content) #5 print(f"Downloaded {image_url} to {target_file}") #6 except MissingSchema: print(f"{image_url} appears to be an invalid URL.") except ConnectionError: raise ConnectionError(f"Could not connect to {image_url}.") get_pictures = PythonOperator( #7 task_id="get_pictures", python_callable=_get_pictures, #8 )
Inspect a complete PythonOperator callable that creates an output directory, reads launch data, extracts image URLs, downloads files, and handles invalid or unreachable URLs. section 2.2.2
