chapter nine

9 Extending Airflow with custom operators and sensors

 

This chapter covers

  • Making your DAGs more modular and concise with custom components
  • Designing and implementing a custom hook
  • Designing and implementing a custom operator
  • Designing and implementing a custom sensor
  • Designing and implementing a custom deferrable sensor
  • Distributing your custom components as a basic Python library

As we’ve seen, one strong feature of Airflow is the ecosystem of operators that allow you to coordinate jobs across many different types of systems. However, at some point, you may want to execute a task on a system that is not supported by Airflow, or you may have a task that you can implement using the PythonOperator but that requires a lot of boilerplate code, which prevents others from easily reusing your code across different DAGs. How should you go about this?

Fortunately, Airflow allows you to easily create new operators for implementing your custom operations. This enables you to run jobs on otherwise unsupported systems or simply to make common operations easy to apply across DAGs. In fact, this is exactly how many of the operators in Airflow were implemented: someone needed to run a job on a certain system and built an operator for it.

We will show you how you can build your own operators and use them in your DAGs. We will also explore how you can package your custom components into a Python package, making them easy to install and reuse across environments.

9.1 Starting with a PythonOperator

9.1.1 Simulating a movie rating API

9.1.2 Fetching ratings from the API

9.1.3 Building the actual DAG

9.2 Building a custom hook

9.2.1 Designing a custom hook

9.2.2 Building our DAG with the MovielensHook

9.3 Building a custom operator

9.3.1 Defining a custom operator

9.3.2 Building an operator for fetching ratings

9.4 Building custom sensors

9.5 Building custom deferrable operator

9.5.1 Executing Asynchronous Tasks Using the Triggerer

9.5.2 Running our Movielens sensor asynchronously

9.6 Packaging your components

9.6.1 Bootstrapping a Python package

9.6.2 Installing your package

9.6.3 Sharing your package with others

9.7 Summary