A data pipeline is an automated sequence that retrieves, transforms, and delivers data between systems. It replaces repeated manual collection with a process that can be rerun, scheduled, and connected to an organization’s data warehouse. section 10.1.2 section 11.2.1
Pipeline design is a stakeholder and business decision as much as a technical one. Evaluate data volume, freshness, expected reuse, available connectors, warehouse access, engineering capacity, and the cost of maintaining automation. Manual downloads may be reasonable for small, static, or short-lived datasets; recurring or high-volume sources usually benefit more from automation. section 10.1.2 section 10.1.3
Common pipeline choices include:
- Use an existing Fivetran connector when one supports the source; connect it with an API key rather than writing extraction code. section 11.2.1
- Build a custom Python extraction and processing workflow, then deploy it through Apache Airflow when no suitable connector exists and regular refreshes are required. section 11.2.1
- Upload relatively static or temporary data manually to an AWS S3 bucket and add it to the warehouse as a static table. section 11.2.1
- Use web scraping when structured API data is unavailable, while planning for brittle scripts, HTML changes, and ongoing maintenance. section 10.2.3
Weekly source refresh
A dataset needed at least weekly is checked for a Fivetran connector. When none exists, the data engineering team prepares a custom Python workflow for Apache Airflow deployment, allowing the source to be refreshed into the warehouse. section 11.2.1
Figure 10.6 If manually downloading and manipulating data frustrates you and creates additional work, you can streamline the process if an API is available.
Use the figure to compare the repeated effort of manual downloading and manipulation with the streamlined workflow enabled by an API pipeline. section 10.1.2
