A pipeline might fetch data from an API, rank the results with Python, and write them to a database. Using a different Airflow operator for each step works, but each operator brings its own interface and implementation to learn and maintain. A generic-operator approach instead aims to run varied tasks through one operator interface. section 11.1 section 11.1.3
Figure 11.1 Illustration of our movie-recommender DAG. The DAG fetches movie recommendations, uses them to rank movies, and stores the result in a database. Each step involves a different operator, adding complexity to the development and maintenance of the DAG.
The three-step pipeline makes the maintenance problem concrete: API access, Python processing, and database storage each use a different operator. section 11.1
An operator implements one piece of work; a DAG coordinates operators, while Airflow tasks manage their scheduling, dependencies, errors, and state. BashOperator and PythonOperator already handle relatively general kinds of work. The stronger generic-operator proposal is to use a single kind of operator across many tasks, with containers providing a way to pursue that goal. section 2.2.1 section 11.1.3
Choose the interface with the maintenance burden in mind:
- Use a system-specific operator when its ready-made integration is valuable. Provider packages supply additional operators; an S3 copy operator, for example, accepts source and destination details and calls the AWS client internally. section 8.1
- Consider one generic operator when many operator interfaces, bugs, and dependency sets make a collection of DAGs difficult to maintain. This shifts attention to how the varied work will run through that shared interface. section 11.1.1 section 11.1.3
- Build a custom operator when an unsupported operation or repeated boilerplate needs a reusable, purpose-built interface across DAGs; that solves a different problem from standardizing every task on one operator. section 9.0 section 9.3
The appeal is operational as well as syntactic: teams learn one operator interface and manage one set of Airflow dependencies for it. That benefit does not make every specialized operator unnecessary; existing integrations can spare you from writing client-level logic. section 11.1.3 section 8.1
