A service is an external software or computing system that participates in an integrated Airflow workload or data operation. Airflow connects to services such as Apache Spark, Google BigQuery, PostgreSQL, and cloud platforms, then orchestrates work between them. section 8.0
Airflow operators provide the integration boundary. They accept connection details and task parameters, translate them into API or client-library operations, and hide much of the service-specific request and response handling. section 8.1
Use services in Airflow to:
- Move data between locations, such as copying an S3 object from one bucket to another with S3CopyObjectOperator. section 8.1
- Delegate computation or queries to systems designed for that workload instead of processing large data sets on Airflow workers. section 12.3.5
- Install provider packages that supply operators and dependencies for AWS, Google Cloud, Azure, PostgreSQL, and other services. section 8.1
Copying an object through a service operator
A task can copy a dated object without directly using the AWS client: S3CopyObjectOperator(task_id="backup", source_bucket_name="databucket", source_bucket_key="/data/{{ ds }}.json", dest_bucket_name="backupbucket", dest_bucket_key="/data/{{ ds }}-backup.json"). section 8.1
Services also shape testing strategy. Tests can mock external calls to avoid dependencies, or emulate services such as PostgreSQL for a more production-like environment; the latter increases setup complexity. section 10.4.1
