A pandas DataFrame is a two-dimensional, labeled table for working with data in Python. In an Airflow pipeline, it can be the working form between reading a file and writing a result: a task loads data, performs tabular operations, then saves the output. section 3.1
Count user events
After an upstream task downloads JSON events, a Python task reads them with pd.read_json(input_path). It groups the DataFrame by date and user, counts rows with .size(), resets the index, and writes the statistics with .to_csv(output_path, index=False). The DataFrame is the working data; the JSON and CSV files are the task’s input and output artifacts. section 3.1
A consistent DataFrame interface also makes transformations reusable across different file formats. One operator can call an input function that returns a DataFrame, an optional transformation function that returns another DataFrame, and an output function that writes it. Taxi CSV and bike JSON can therefore follow the same task pattern while using different readers and transformations. Changing formats still requires the appropriate read or write callable; the DataFrame does not remove those differences. section 13.4
Listing 13.14 Example function transforming taxi data
def transform_taxi_data(df): #1
return (
df
.assign(
starttime=lambda df: pd.to_datetime(df.pickup_datetime),
stoptime=lambda df: pd.to_datetime(df.dropoff_datetime),
tripduration=lambda df: (df.stoptime –
➥ df.starttime).dt.total_seconds().astype(int),
)
.rename(columns={"pickup_locationid":
➥ "start_location_id","dropoff_locationid": "end_location_id"})
➥ .drop(columns=["trip_distance", "pickup_datetime",
➥ "dropoff_datetime"])
) #2
The taxi transformation shows the DataFrame-in, DataFrame-out contract: assign calculated time fields, rename location columns, and drop unneeded columns before returning the result. section 13.4
Combine recipe files for preprocessing
A daily preprocessing task reads individual recipe JSON files and builds one DataFrame of chunks. Each recipe contributes an ingredients chunk and an instructions chunk, both carrying its name. A subsequent DataFrame transformation strips text and assigns UUIDs before the processed data is saved to MinIO. Airflow runs this work through a DockerOperator, keeping the processing code in a container. section 14.5
