concept volume in category apache airflow

appears as: volumes, volume, volume
Data Pipelines with Apache Airflow

This is an excerpt from Manning's book Data Pipelines with Apache Airflow.

The container function in pytest_docker_tools is also a fixture, so that too can only be called by providing it as an argument to a test. It takes a number of arguments which configure the container to start, in this case the image id which was returned from the fetch() fixture, and the ports to expose. Just like running Docker containers on the command line, we could also configure environment variables, volumes and more.

Besides orchestrating the running of pods across the different worker nodes, Kubernetes also provides built-in functionality for managing secrets and storage. In essence, this means that we can, for example, request a storage volume from the Kubernetes master and mount this volume as persistent storage inside the container. As such, these storage volumes function in a similar manner as the Docker volume mounts we saw in the previous section, but are managed by Kubernetes. This means that we don’t have to worry about where the storage comes from (unless you are responsible for operating the cluster of course), but can simply request and use the volume provided by Kubernetes.

Besides these arguments, we also supply two extra arguments, volumes and volume_mounts, which specify how the volumes we created in the previous section should be mounted into the tasks Kubernetes pod. These configuration values are created using two config classes built into Airflow - Volume and VolumeMount:

Listing 11.32.
from airflow.contrib.kubernetes.volume import Volume
from airflow.contrib.kubernetes.volume_mount import VolumeMount
 
volume = Volume( #A
    name="data-volume",
    configs={"persistentVolumeClaim": {"claimName": "data-volume"}}
)
volume_mount = VolumeMount( #B
    "data-volume",
    mount_path="/data", #C
    sub_path=None,
    read_only=False, #D
)

Here, we first create a Volume configuration object, which references the persistent volume claim data-volume, which we created as a Kubernetes resource in the previous section. Next, we create a VolumeMount configuration object, which refers to the volume configuration we just created (‘data-volume’) and specifies where this volume should be mounted within the pods container. These configuration objects can then be passed to the KubernetesPodOperators using the volumes and volume_mounts arguments.