Atomicity means treating related operations as an indivisible unit: everything completes successfully, or nothing changes. In Airflow, an atomic task either produces its proper result or fails without leaving misleading partial state. section 3.7.1
A task that calculates user statistics, writes them to a file, and then sends an email is non-atomic. If the email service fails, the task is marked failed even though the statistics file already exists. Separate calculation from notification so each task reports its own outcome. section 3.7.1
Design atomic Airflow work by considering both task boundaries and operational dependencies:
- Split independent operations when a failure in one should not invalidate the result of another. section 3.7.1
- Keep tightly coupled steps together when one operation requires state created immediately by another, such as authentication followed by an API request. section 3.7.1
- Wrap multi-statement database changes in a transaction so deletion and replacement either both succeed or neither does. section 13.6
Replacing interval-specific database rows
A data pipeline can delete existing rows for an Airflow execution date and insert the refreshed DataFrame rows inside one database transaction. If either operation fails, the transaction leaves no incomplete replacement behind. section 13.6
