Apache Spark is a distributed data-processing engine for workloads too large or resource-intensive for the machine running Airflow. A strong separation keeps Airflow responsible for starting and monitoring a job while Spark performs the actual processing on dedicated machines. section 8.3.3 section 13.4
Choose an Airflow launch path according to where Spark is available and what configuration the Airflow environment can access:
- Use SparkSubmitOperator when the Airflow machine has the spark-submit binary and YARN client configuration needed to locate the Spark instance. section 8.3.3
- Use SSHOperator when Airflow can connect securely to a Spark machine but does not have Spark client configuration locally. section 8.3.3
- Use SimpleHTTPOperator when Livy, Spark's REST API, is running and exposes the job submission endpoint. section 8.3.3
- For smaller datasets, use pandas through PythonOperator instead; unlike Spark, that processing consumes the Airflow machine's resources. section 13.4
Citi Bike and Yellow Cab transformations
After downloading Citi Bike and Yellow Cab data, trigger a Spark job with SparkSubmitOperator or another Airflow operator. The job reads the datasets from S3, transforms station coordinates and taxi zones for accurate comparison, and writes the results back to S3. For data that fits on one machine, pandas is the smaller-scale alternative. section 13.4
