A REST API exposes data and operations through HTTP requests. In an Airflow pipeline, it can provide an external event or dataset that a workflow retrieves, transforms, and loads alongside files, databases, and object storage. section 7.0 section 13.1
A request targets an endpoint and may include authentication and path parameters. The response can contain structured records, such as ride locations and timestamps, for downstream processing. section 13.2.2
Requesting recent Citi Bike rides
The command `curl --user citibike:cycling http://localhost:8082/recent/hour/1` requests rides finished during the past hour. Changing the endpoint to `/recent/day/3` requests rides from the past three days. section 13.2.2
Account for practical API constraints when designing an extraction task:
- Authenticate each request with the credentials required by the service. section 13.2.2
- Choose a request interval that retrieves all records without exceeding response-size or result-count limits. section 13.2.2
- Pass the returned records into transformations and storage components in the surrounding data pipeline. section 13.1
Figure 13.1 Our Docker Compose file creates several services. Our task is to load data from the REST API and file share and then transform it so we can eventually view the fastest method of transportation on the resulting web page.
Use the system diagram to place a REST API alongside file shares, databases, object storage, and a web application in a complete pipeline. section 13.1
