chapter seven

7 Building Custom Components

 

This chapter covers:

  • How to make your DAGs more modular and succinct by implementing custom components for interacting with (remote) systems.
  • How to implement a custom hook and how to use this hook to interact with an external system.
  • How to design and implement your own custom operator to perform a specific task.
  • How to design and implement your own custom sensor.
  • How to distribute your custom components as a basic Python library.

One of the strong features of Airflow is that it was designed to be easily extendable to support coordinating jobs across many different types of systems. We have already seen some of this functionality in earlier chapters, where we were able to execute a spark job on a Spark cluster using the SparkSubmitOperator, but you can (for example) also use Airflow to run jobs on an ECS (Elastic Container Services) cluster in AWS using the EcsOperator, to perform queries on a Postgres database using the PostgresOperator, and much more.

However, at some point you may run into the issue that you want to execute a task on a system that is not supported by Airflow. Or you may have a task that you can implement using the PythonOperator, but requires a lot of boilerplate code which prevents others from easily reusing your code across different DAGs. How should you go about this?

7.1       Starting with a PythonOperator

7.1.1   Simulating a movie rating API

7.1.2   Fetching ratings from the API

7.1.3   Building the actual DAG

7.2       Building a custom hook

7.2.1   Designing a custom hook

7.2.2   Building our DAG with the Movielens hook

7.3       Building a custom operator

7.3.1   Defining a custom operator

7.3.2   Building an operator for fetching ratings

7.4       Building custom sensors

7.5       Packaging your components

7.5.1   Bootstrapping a Python package

7.5.2   Installing your package

7.6       Summary