A workflow that collects user events needs a defined way to ask for records and interpret the result. An API supplies that interface: a request to an event endpoint returns JSON that a downstream task can analyze. The same idea applies when software requests cloud-service operations. section 3.1 section 8.1
The interface serves different jobs in an Airflow workflow:
- Ingest data: use curl to fetch events, then let a Python task read the result and calculate statistics. section 3.1
- Start work: POST to Airflow’s REST API to trigger a DAG run, optionally supplying configuration that its tasks can read. section 7.2
- Perform external operations: use a provider operator to pass required arguments to a cloud SDK rather than handling its request details directly. section 8.1
Listing 13.4 Sample request to the Citi Bike REST API
$ date
Thu 19 Sep 2024 20:31:07 CEST
$ curl --user citibike:cycling
➥ http://localhost:8082/recent/hour/1 #1
[
{
"end_station_id": "5746.02", #2
"end_station_latitude": 40.7667405590595, #2
"end_station_longitude": -73.9790689945221, #2
"end_station_name": "7 Ave & Central Park South", #2
"start_station_id": "6046.02", #2
"start_station_latitude": 40.77492513, #2
"start_station_longitude": -73.98266566, #2
"start_station_name": "W 67 St & Broadway", #2
"starttime": "Thu, 19 Sep 2024 19:21:35 GMT", #2
"stoptime": "Thu, 19 Sep 2024 19:27:08 GMT", #2
"tripduration": 333 #2
},
{
"end_station_id": "5414.06",
"end_station_latitude": 40.711066,
"end_station_longitude": -74.009447,
"end_station_name": "Fulton St & Broadway",
"start_station_id": "6046.02",
"start_station_latitude": 40.692418292578466,
"start_station_longitude": -73.98949474096298,
"start_station_name": "Fulton St & Adams St",
"starttime": "Thu, 19 Sep 2024 19:18:10 GMT",
"stoptime": "Thu, 19 Sep 2024 19:30:33 GMT",
"tripduration": 744
},
...
]
The authenticated Citi Bike request makes the input and output concrete: the URL selects a time window, while each returned record supplies ride times and station coordinates. section 13.2.2
Listing 7.7 Triggering a DAG using the Airflow REST API
# URL is /api/v1
curl \
-u airflow:airflow \ #1
-X POST \
"http://localhost:8080/api/v1/dags/11_inspect_dag_run_config/dagRuns" \
-H "Content-Type: application/json" \
-d '{"conf": {}}' #2
{
"conf": {},
"dag_id": "11_inspect_dag_run_config",
"dag_run_id": "manual__2024-04-20T08:10:59.817154+00:00",
"data_interval_end": "2024-04-20T00:00:00+00:00",
"data_interval_start": "2024-04-19T00:00:00+00:00",
"end_date": null,
"execution_date": "2024-04-20T08:10:59.817154+00:00",
"external_trigger": true,
"last_scheduling_decision": null,
"logical_date": "2024-04-20T08:10:59.817154+00:00",
"note": null,
"run_type": "manual",
"start_date": null,
"state": "running"
}
curl \
-u airflow:airflow \
-X POST \
"http://localhost:8080/api/v1/dags/11_inspect_dag_run_config/dagRuns" \
-H "Content-Type: application/json" \
-d '{"conf": {"supermarket": 1}}'
{
"conf": {
"supermarket_id": 1
},
"dag_id": "11_inspect_dag_run_config",
"dag_run_id": "manual__2024-04-20T08:10:46.623540+00:00",
"data_interval_end": "2024-04-20T00:00:00+00:00",
"data_interval_start": "2024-04-19T00:00:00+00:00",
"end_date": null,
"execution_date": "2024-04-20T08:10:46.623540+00:00",
"external_trigger": true,
"last_scheduling_decision": null,
"logical_date": "2024-04-20T08:10:46.623540+00:00",
"note": null,
"run_type": "manual",
"start_date": null,
"state": "running"
}
This POST shows the other direction of integration: an external caller sends JSON to Airflow and receives details of the resulting DAG run. section 7.2
Check operational limits before relying on an API for ingestion. The event service retains data for only 30 days, so longer history must be stored elsewhere. The Citi Bike example has no request-size limit, but a real API might cap results; requests would then need to run often enough to collect every record. Provider operators simplify cloud calls, though their interfaces and bugs become another maintenance concern. section 3.1 section 13.2.2 section 11.1.1
