S3CopyObjectOperator

As explained in Data Pipelines with Apache Airflow, Second Edition

S3CopyObjectOperator is an Airflow operator that copies an object from one Amazon S3 bucket or location to another.

S3CopyObjectOperator is an Airflow task for copying one object between Amazon S3 buckets or locations. It wraps the AWS SDK, so the task requires bucket names and object keys rather than direct boto3 code. section 8.1

Configure the operator with the source and destination details:

• Install the Amazon provider package: `apache-airflow-providers-amazon`. section 8.1
• Set `source_bucket_name` and `source_bucket_key` for the object to copy. section 8.1
• Set `dest_bucket_name` and `dest_bucket_key` for the copied object. section 8.1
• Use Airflow templating in keys when the destination depends on the execution date, such as `/data/{{ ds }}-backup.json`. section 8.1

Copy an object into a processing bucket

from airflow.providers.amazon.aws.operators.s3 import S3CopyObjectOperator copy_mnist = S3CopyObjectOperator( task_id="download_mnist_data", source_bucket_name="sagemaker-sample-data-eu-west-1", source_bucket_key="algorithms/kmeans/mnist/mnist.pkl.gz", dest_bucket_name="[your-bucket]", dest_bucket_key="mnist.pkl.gz", ) In a larger pipeline, this task runs first to place public MNIST data in an owned bucket for later processing. The operator internally calls boto3 `copy_object()`. section 8.2.2 section 8.1

Figure 8.1 An Airflow operator translates given arguments to operations on the Cloud SDK.

Use the diagram to follow how operator arguments become AWS Cloud SDK operations. section 8.1

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes

Explains configuration and the boto3 `copy_object()` implementation while omitting irrelevant operator arguments.

Examples

Uses dated backup keys and copies the public MNIST object into an owned bucket.

What it assumes

Requires the Amazon provider package and configured access to the source and destination S3 locations.

What you can do after

Apply it via `S3CopyObjectOperator` to configure an Airflow task that copies a selected S3 object.

How it connects

Connects Airflow tasks and DAGs with Amazon S3, AWS provider packages, boto3, and downstream SageMaker processing.

liveBookmanning.com

Read Data Pipelines with Apache Airflow, Second Edition on liveBook Get the book at manning.com

Related concepts

• operatorkind of · An Airflow operator is a reusable workflow component that defines and executes a task through a standard operator interface.
• datasetuses · A dataset is an organized collection of related data items, records, or observations treated as a unit for storage, processing, analysis, or computation.
• object storeuses · An object store is a storage system that stores and retrieves data as independently addressable objects containing content and associated metadata.

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.