When a pipeline task needs to copy data from S3, test that task on its own before running the whole DAG. `airflow tasks test` executes one task for a specified execution date or run ID and displays its logs. It does not register task records in the Airflow metastore, but an external-system task can still perform real work. section 8.2.3 section 10.3
For a local AWS test:
- Make cloud credentials available to the task; the example selects a shared-credentials profile with `AWS_PROFILE`. section 8.2.3
- Set `AIRFLOW_HOME`, point `AIRFLOW__CORE__DAGS_FOLDER` at the DAGs if necessary, and run `airflow db init`. section 8.2.3
- Run the command with the DAG ID, task ID, and a test date. Choose the date carefully if a destination key uses a template such as `{{ ds }}`. section 8.2.3
Test a download task
Run `airflow tasks test 01_aws_handwritten_digits_classifier download_mnist_data 2024-01-01`. The example runs an S3 copy task, prints its execution logs, and copies the data into the configured bucket. To inspect Python code during a task test, insert `import pdb; pdb.set_trace()` and use the command-line debugger when execution reaches it. section 8.2.3 section 10.3
Listing 8.4 Setting up to locally test AWS operators
# Add secrets in ~/.aws/credentials: # [myaws] # aws_access_key_id=AKIAEXAMPLE123456789 # aws_secret_access_key=supersecretaccesskeydonotshare!123456789 export AWS_PROFILE=myaws export AWS_DEFAULT_REGION=eu-west-1 export AIRFLOW_HOME=[your project dir] airflow db init #1 airflow tasks test \ #2 01_aws_handwritten_digits_classifier \ #3 download_mnist_data \ #4 2024-01-01 #5
The setup listing puts the environment variables, database initialization, and single-task command together. Its credentials are illustrative; production credentials should be managed with a secret manager. section 8.2.3
