liveBook · Manning

Volume

As explained in Data Pipelines with Apache Airflow, Second Edition

A Kubernetes volume is a storage abstraction that makes data available to containers in a pod.

A Kubernetes volume makes storage available inside a pod’s containers. A PersistentVolume provides storage, and a PersistentVolumeClaim requests it; pods can use the claim to store task results. Mounting the volume makes that storage accessible at a path in the container. section 11.5.2 section 11.5.1

Passing task data through a mounted volume

In a MovieLens workflow, one KubernetesPodOperator task writes fetched ratings to /data/ratings, and a later task reads them from the same mounted volume to produce rankings. section 11.5.3

Listing 11.24 Volumes and volume mounts
from kubernetes.client import models as k8s

...

volume_claim = k8s.V1PersistentVolumeClaimVolumeSource(   #1
    claim_name="data-volume"
)
volume = k8s.V1Volume(                       #2
    name="data-volume",
    persistent_volume_claim=volume_claim
)
volume_mount = k8s.V1VolumeMount(
    name="data-volume",
    mount_path="/data",                       #3
    sub_path=None,
    read_only=False,                          #4
)

See how a persistent volume claim is referenced by a V1Volume and mounted at /data with a V1VolumeMount. section 11.5.3

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Defines storage resources in YAML and shows how Python SDK objects connect a claim to a pod mount.
Examples
Uses a MovieLens ratings-and-rankings workflow to pass files between task containers through /data.
What it assumes
Assumes a functioning Kubernetes cluster and familiarity with creating resources and running Airflow Kubernetes tasks.
What you can do after
Configure persistent storage for Kubernetes task containers using a claim and a mounted path.
How it connects
Connects PersistentVolumes and PersistentVolumeClaims to Kubernetes pods and the KubernetesPodOperator.

Also explained in

Compare how 8 books explain volume →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.