PandasOperator is a custom Airflow operator for reusable, single-machine pandas pipelines. It separates reading, transformation, and writing into configurable callables, allowing similar tasks to share execution logic while varying datasets, formats, and paths. section 13.4 section 13.7
Processing taxi data
A taxi task can read CSV objects from a datalake bucket, use transform_taxi_data to parse pickup and dropoff timestamps, calculate trip duration, rename location columns, and remove unused fields, then write Parquet output under a date-templated path. The same operator structure can process Citi Bike data by changing its input, transformation, and output arguments.
The operator suits data that fits on the Airflow machine. pandas execution consumes that machine’s resources and can threaten Airflow stability when a job reaches resource limits. Larger workloads can instead use SparkSubmitOperator or a containerized option such as KubernetesPodOperator, leaving Airflow to start and monitor the external job. section 13.4
