12 Best practices
This chapter covers
- Writing clean, understandable DAGs using style conventions
- Using consistent approaches for managing credentials and configuration options
- Generating repeated DAGs and tasks using factory functions
- Designing reproducible tasks by enforcing idempotency and determinism constraints
- Handling data efficiently by limiting the amount of data processed in your DAG
- Using efficient approaches for handling/storing (intermediate) data sets
- Managing concurrency using resource pools
Previously, we have described most of the basic elements that go into building and designing data processes using Airflow DAGs. Now, we dive a bit deeper into some best practices that can help you write well-architected DAGs that are both easy to understand and efficient in terms of how they handle your data and resources.
12.1 Writing clean DAGs
Writing DAGs can easily become a messy business. For example, DAG code can quickly become overly complicated or difficult to read—especially if DAGs are written by team members with very different styles of programming. In this section, we touch on some tips to help you structure and style your DAG code, hopefully providing some (often needed) clarity for your intricate data processes.