A shared drive is a service that makes files available to other systems through a network or shared-storage interface. In this workflow, a file share exposes Yellow Cab ride data as downloadable CSV files. section 13.1 section 13.2.1
A file share in this workflow has several operational characteristics:
- Exposes an index that lists available files rather than requiring filenames to be known in advance. section 13.3.2 section 13.2.1
- Serves static CSV files containing taxi rides, timestamps, location identifiers, and trip distances. section 13.2.1
- Retains only one full hour of data and removes older data automatically. section 13.2.1
- Requires custom pipeline behavior when built-in Airflow operators do not match its listing and download requirements. section 13.3.2
Copying files into object storage
An Airflow Python task requests the file-share index, downloads each listed CSV with HTTP, and uploads it to MinIO under raw/taxi/. Because the upload operation fails when an object already exists, the task catches ValueError and continues. section 13.3.2
Listing 13.9 Downloading data from the Yellow Cab file share to MinIO storage
def _download_taxi_data():
taxi_conn = BaseHook.get_connection(conn_id="taxi")
s3_hook = S3Hook(aws_conn_id="s3")
url = f"http://{taxi_conn.host}"
response = requests.get(url) #1
files = response.json()
for filename in [f["name"] for f in files]:
response = requests.get(f"{url}/{filename}") #2
s3_key = f"raw/taxi/{filename}"
s3_hook.load_string(string_data=response.text, key=s3_key,
➥ bucket_name="datalake") #3
)
download_taxi_data = PythonOperator(
task_id="download_taxi_data",
python_callable=_download_taxi_data,
)
Use the annotated listing to trace file discovery, HTTP downloads, object-key construction, and MinIO uploads in one Airflow task. section 13.3.2
