Airflow deployment

As explained in Data Pipelines with Apache Airflow, Second Edition

An Airflow instance is an independently configured deployment of the Airflow platform with its own components and operational environment.

An Airflow deployment is an independently configured operational environment for running DAGs. At organizational scale, teams can share one instance or operate separate instances with isolated resources, permissions, failures, and upgrade decisions. section 15.9

Choose the deployment boundary by weighing these operational trade-offs:

• Share one instance to maximize resource use and potentially reduce cost, especially when teams' scheduling demands do not overlap. section 15.9
• Expect shared deployments to complicate resource contention, DAG ownership, permissions, upgrades, and failure isolation. section 15.9
• Give each team an instance for stronger isolation and independent configuration, while accepting the overhead of operating multiple environments. section 15.9

A deployment with multiple executors

Configure CeleryExecutor and KubernetesExecutor together through the Apache Airflow Helm Chart. Run ordinary tasks with the default image, Kubernetes tasks with a TensorFlow image, and override an individual task with another image. A missing TensorFlow dependency can make the Celery task fail while the Kubernetes tasks succeed in their tailored images. section 17.2.9

Figure 15.18 Two teams with their own Airflow instances. Each team could configure its own Airflow instance. Team 1 could use the KubernetesExecutor instead of the CeleryExecutor without affecting team 2, for example.

Compare two team-owned Airflow instances and see how one team can change its executor without affecting the other. section 15.9

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes

Compares shared and separate instances, then demonstrates executor and image configuration without detailing every deployment component.

Examples

Uses shared-team and team-owned instances plus a three-task DAG that contrasts CeleryExecutor and KubernetesExecutor images.

What it assumes

Assumes familiarity with DAGs and executors, while demonstrating their deployment-level configuration and operational consequences.

What you can do after

Operate/configure it via the Apache Airflow Helm Chart with multiple executors and task-specific container images.

How it connects

Connects DAGs, executors, role-based access control, Docker images, Helm, Kubernetes, and resource isolation.

liveBookmanning.com

Read Data Pipelines with Apache Airflow, Second Edition on liveBook Get the book at manning.com

Related concepts

• Kubernetesuses · Kubernetes is an open-source platform for orchestrating and automating the deployment, scaling, and management of containerized workloads across computing clusters.
• Docker Composeuses · Docker Compose is a tool for defining and running applications composed of multiple containers through declarative configuration.
• values.yaml fileconfigured by · A values.yaml file is a Helm configuration file containing parameter values used to customize chart templates and their deployed resources.
• container imageuses · A container image is a packaged filesystem and configuration artifact from which a container can be created and run.
• Helmconfigured by · Helm is a Kubernetes package manager and templating tool that defines, packages, installs, upgrades, and manages applications through charts.

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.