liveBookmanning.com

What you will learn in Data Pipelines with Apache Airflow, Second Edition

Orchestration for data and AI

by Julian de Ruiter, Ismael Cabral, Kris Geusebroek, Daniel van der Ende, Bas Harenslak

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• Apache AirflowApache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows, especially data pipelines.
• container imageA container image is a packaged filesystem and application specification from which a container can be created.
• executorAn Airflow executor is an execution framework component that determines how workflow tasks are dispatched and run by workers.
• containerA Docker container is an isolated runtime instance that packages an application and its dependencies using Docker.
• directed acyclic graphA directed acyclic graph is a directed graph with no cycles, often used to represent workflow tasks and their dependencies.
• retrieval-augmented generationA RAG workflow is a retrieval-augmented generation workflow that retrieves relevant information and supplies it to a generative model for producing an output.
• CeleryExecutorCeleryExecutor is an Airflow executor that distributes task execution across worker processes and machines using Celery.
• callbackA callback is a function supplied to another component so that it is invoked when a specified event occurs.
• vector databaseA vector database is a database that stores vector representations and supports operations such as similarity search and vector-based retrieval.
• KubernetesExecutorKubernetesExecutor is an Airflow executor that runs task execution in Kubernetes-managed workloads across machines.
• dependencyA software dependency is an external package or component required by software for installation or execution.
• schedulingA schedule is a rule that specifies when a workflow should run.
• operatorA generic operator is an Airflow operator designed to run varied task types through one common operator interface.
• workerAn Edge Worker is a worker process that registers with an Airflow API server and executes tasks assigned to it remotely.
• sensorAn Airflow sensor is an operator that repeatedly checks an external condition and succeeds when the condition becomes true or fails after a timeout.
• DockerOperatorDockerOperator is an Airflow operator that starts and manages Docker containers for task execution.
• LocalExecutorLocalExecutor is an Airflow executor that runs tasks locally on a single machine, allowing local parallel execution.
• schedulerThe scheduler is an Apache Airflow component that determines when eligible workflow tasks should be scheduled for execution.
• migrationExtract, transform, load is a data-integration process that retrieves data from a source, transforms it, and loads the result into a destination.
• deferrable operatorA deferrable operator is an Airflow operator that yields its worker while waiting and resumes when an asynchronous trigger signals that work is ready to continue.
• PythonOperatorPythonOperator is an Airflow operator that executes a Python callable as a task.
• EdgeExecutorEdgeExecutor is an Airflow executor that assigns tasks to workers running outside the Airflow instance environment.
• pipelineA data workflow is an ordered set of data-processing activities and their dependencies.
• asset eventAn asset event is a record that an Airflow task has successfully produced or updated an asset and that can trigger dependent DAG runs.
• packageA software package is a distributable bundle of code and metadata that can be installed into a software environment.
• pytestpytest is a Python testing framework that discovers and runs tests and provides fixtures and extensibility mechanisms.
• Airflow API serverThe Airflow API server is the service interface for viewing workflows, monitoring runs, and mediating database access for Airflow components.
• data frameA pandas DataFrame is a two-dimensional labeled tabular data structure provided by the pandas library.
• taskAn Airflow task is a wrapper that manages the execution, dependencies, scheduling, and error handling of an operator.
• consumer DAGA consumer DAG is an Airflow DAG whose execution is triggered by events indicating that one or more input assets have changed or become available.
• trigger ruleA trigger rule is a condition that determines whether an Airflow task is ready to execute based on the execution status of its dependencies.
• PrometheusPrometheus is an open-source monitoring and alerting system that collects time-series metrics from instrumented or exported endpoints.
• universally unique identifierA UUID is a universally unique identifier represented as a fixed-format value intended to identify an entity with extremely low collision probability.
• embeddingA vector is an ordered numerical quantity representing a point, direction, or set of measurable attributes in a multidimensional space.
• vectorizationVectorization is the transformation of content into numerical vectors that capture features for computational comparison or retrieval.
• assetAn asset is a virtual reference identified by a string ID or URI that represents a data object or other consumable or producible resource.
• command-line interfaceA command-line interface is a user interface in which commands and arguments are provided through a text-based shell.
• Airflow connectionAn Airflow connection is a stored configuration record containing the information and credentials needed to access an external system.
• incremental processingAn incremental data pipeline processes only data added or changed since a previous processing interval rather than reprocessing the complete dataset.
• producer DAGA producer DAG is a directed acyclic graph whose tasks prepare or generate a data asset for downstream workflows.
• idempotencyAn idempotent task produces the same effective result when rerun with the same inputs and state.
• asset-based scheduleAsset-aware scheduling is a workflow scheduling technique that triggers work based on the availability or update state of data assets rather than only on fixed times.
• message brokerA broker is a message-queue component that receives, stores, and delivers messages between producers and consumers.
• logical dateA logical date is the date or time label associated with the data interval or temporal point represented by a DAG run.
• dag processorThe DAG processor manager is an Airflow internal component that coordinates parsing of DAG files and aggregates information about the parsed files.
• parallelismExecution of independent tasks concurrently rather than one after another.
• language modelA large language model is a machine learning model trained on extensive text data to understand and generate natural-language text.
• PostgresHookPostgresHook is an Airflow hook for communicating with PostgreSQL databases.
• airflow tasks testairflow tasks test is an Airflow command-line operation that executes and verifies a single task for a specified execution date or run identifier without recording task state.
• object storeObject Storage is an Airflow feature that provides a unified syntax for interacting with object-storage systems and transferring data between them.
• XComXCom is an Apache Airflow mechanism for storing and exchanging small task results through the Airflow metastore.
• secret managementA secrets manager is an external system that securely stores and provides sensitive values such as credentials and keys.
• templating languageTemplating is the technique of generating a command or other text by substituting runtime values into a predefined template.
• text chunkA chunk is a smaller piece of source text created by dividing a larger document for retrieval and processing.
• triggererThe triggerer is an Apache Airflow component that monitors conditions for tasks performing asynchronous processing.
• SQLExecuteQueryOperatorSQLExecuteQueryOperator is an Airflow operator that executes SQL queries against a database.
• triggerAn Airflow trigger is an asynchronous Python component that waits for an external condition and signals a deferred task when the condition is met.
• max active tasksMax active tasks is an Airflow DAG setting that limits the number of tasks from that DAG that may run simultaneously.
• code executionAn execute method is the Airflow operator method that performs the operator's work during task execution.
• DockerfileA Dockerfile is a text file containing instructions for building a Docker image.
• test fixtureA test fixture is a reusable test setup component that provides resources or state to a test and may perform cleanup afterward.
• testA test function is a callable function whose execution verifies expected software behavior.
• Airflow metastoreThe Airflow metastore is the metadata database that stores serialized workflow definitions, execution state, and stored connection information.
• backfillingBackfilling is the recomputation or execution of previously scheduled workflow periods after data or processing logic changes.
• contractAn API is a defined interface through which software applications programmatically access data or functionality.
• branchBranching is the conditional selection of one of multiple execution paths within a workflow task or structure.
• catchupCatchup is an Airflow scheduling setting that creates runs for past schedule intervals that were missed or not previously executed.
• failureAn error is a failure condition or abnormal event reported during system operation.
• volumeA Kubernetes volume is a storage abstraction that makes data available to containers in a pod.
• mockingA mock is a test double that replaces a dependency with controlled behavior for verifying software interactions.
• DAG fileA DAG file is a Python script that declares the structure, tasks, dependencies, and metadata of an Airflow directed acyclic graph.
• Taskflow APIThe Taskflow API is an Apache Airflow interface for defining workflows by expressing task logic and dependencies through Python functions.
• cachingCaching is the technique of storing computed or retrieved objects so later requests can reuse them without repeating the original work.
• monitoringLogging is the recording of operational events and messages produced during software execution.
• data interval startThe data interval start is an Airflow scheduling context value containing the beginning of a DAG run's data interval.
• FileSensorFileSensor is an Airflow sensor that checks whether a file exists at a specified path.
• BashOperatorBashOperator is an Airflow operator that executes a Bash command or script as a task.
• start dateThe start_date parameter specifies the starting date and time from which an Airflow schedule is computed.
• JinjaJinja-based templating is the generation of text by evaluating expressions and substitutions embedded in a Jinja template.
• static analysisStatic code analysis is the examination of source code without executing it to detect defects, style violations, or other issues.
• podA Kubernetes pod is the smallest deployable unit in Kubernetes and contains one or more containers sharing networking and storage context.
• GrafanaGrafana is an open-source observability platform for querying, visualizing, and monitoring data from multiple sources.
• agent orchestrationA container orchestration system is software that coordinates the deployment, execution, scaling, and distribution of containers across multiple machines.
• serviceAn external system is a software or computing system outside Airflow that participates in an integrated workload or data operation.
• KubernetesA Kubernetes cluster is a set of machines managed by Kubernetes that provides pooled resources for running containerized workloads.
• all_successall_success is an Airflow trigger rule requiring every dependency of a task to complete successfully before the task can execute.
• queueA queue is an ordered data structure from which work items are retrieved for processing.
• containerized taskA containerized task is a task executed within an isolated software container.
• secrets backendAn Airflow secrets backend is an integration component that retrieves sensitive configuration from an external secrets-management system for use in Airflow.
• intervalA data interval is the bounded period of data associated with one scheduled DAG run.
• dynamic task mappingDynamic task mapping is an Apache Airflow feature that creates task instances at runtime from iterable input values for a DAG run.
• right-shift operationThe right shift operator is a Python operator that shifts the binary representation of an integer to the right and is overloaded by Airflow for expressing task dependencies.
• pytest-docker-toolspytest-docker-tools is a Python package that provides pytest fixtures and helper functions for managing Docker containers in tests.
• dag integrity testA DAG integrity test is a test that validates the structural correctness of Airflow DAG definitions, including acyclicity and unique task identifiers.
• encryptionEncryption is the transformation of readable data into ciphertext that requires a cryptographic key for recovery.
• MinIOMinIO is an object-storage server that provides an Amazon S3-compatible API.
• style guideA style guide is a documented set of conventions for writing source code, including formatting, naming, and layout rules.
• formattingA code formatter is a software tool that rewrites source code into a consistent formatting style according to configured or built-in rules.
• RuffRuff is a fast Python linter and code formatter implemented in Rust.
• task contextThe Airflow context is the runtime collection of metadata and parameters made available during task execution.
• HTTPSHTTPS is the secure form of HTTP that uses Transport Layer Security to authenticate endpoints and encrypt communication.
• private keyA private key is a secret cryptographic key used to decrypt data or create digital signatures.
• certificateA digital certificate is a signed data structure that binds an identity to a public key and includes information for validating that binding.
• performanceA metric is a quantified measurement used to characterize or analyze data or system behavior.
• text-embedding-3-largetext-embedding-3-large is OpenAI's text embedding model for generating numerical vector representations of content.
• SparkApache Spark is an open-source distributed computing engine for large-scale data processing and analytics.
• docker builddocker build is a Docker CLI command that constructs an image from a Dockerfile and its build context.
• BranchPythonOperatorBranchPythonOperator is an Airflow operator that selects downstream task paths by evaluating a Python callable that returns task identifiers.
• poolAn Airflow pool is a named set of execution slots that limits the concurrency of tasks competing for a resource.
• DagBagDagBag is an Airflow object that loads and parses DAG definitions from Python files while collecting import errors.
• timetableAn Airflow timetable is a scheduling abstraction that determines the times at which DAG runs are created.
• Airflow variableAn Airflow variable is a named global value stored in Airflow for use by workflows and tasks.
• RESTA REST API is an HTTP-based application programming interface that exposes resources and operations through stateless web requests.
• shared driveA file share is a service that makes files available for access by other systems over a network or shared storage interface.
• Docker ComposeDocker Compose is a tool for defining and managing multicontainer Docker applications using configuration files.
• kubectlkubectl is the command-line client for interacting with Kubernetes clusters and resources.
• paginationPagination is the technique of retrieving a large result set through multiple bounded pages of records.
• Lightweight Directory Access ProtocolAn LDAP service is a directory service that supports queries through the Lightweight Directory Access Protocol.
• downstream taskA downstream task is a workflow task that follows another task in the dependency structure and may depend on its completion.
• python_callablepython_callable is a PythonOperator argument that specifies the callable to execute.
• task groupAn Airflow task group is a grouping abstraction that organizes related tasks and presents them as a condensed unit in the workflow interface.
• attributeAn attribute is a named property of a directory entry that stores a value describing the entity.
• grid viewA grid view is an Airflow interface view that presents historical and active runs of a DAG together with task outcomes.
• expand()expand() is an Airflow task-mapping method that specifies the iterable values for which separate mapped task instances are created.
• mapped task instanceA mapped task instance is an execution-specific occurrence of an Airflow task created for one element of dynamic task-mapping input.
• partial applicationpartial() is an Airflow task-mapping method that specifies operator parameters kept constant across mapped task instances.
• asynchronous programmingAsynchronous processing is execution in which a task can progress or await conditions without continuously occupying a worker.
• atomicityAn atomic transaction is an indivisible database transaction whose operations either all take effect or have no effect.
• schemaA database schema is the structural definition of a database's tables, columns, relationships, and other database objects.
• pytest-mockpytest-mock is a Python package that provides pytest integration and a convenience wrapper for mock objects.
• poke modePoke mode is a sensor execution mode in which the sensor retains its task slot while waiting between condition checks.
• reschedule modeReschedule mode is a sensor execution mode in which the sensor releases its task slot after checking the condition and reacquires capacity for later checks.
• sensor deadlockSensor deadlock is a condition in which polling sensors exhaust available task capacity and prevent other tasks from running.
• task slotA worker slot is a unit of task-execution capacity that can be occupied by a running Airflow task.
• CronTriggerTimetableCronTriggerTimetable is an Airflow timetable class that creates DAG runs at cron-specified trigger times without representing the schedule as data intervals.
• async/awaitAn asynchronous method is a callable method whose execution can suspend while awaiting an operation and allow other asynchronous work to proceed.
• defer methodThe defer method is an Airflow operator method that transfers waiting task execution to the triggerer.
• execute_complete methodThe execute_complete method is an Airflow operator callback that handles completion after a deferred task's trigger signals success.
• data interval endThe data interval end is an Airflow scheduling context value containing the end of a DAG run's data interval.
• BaseHookBaseHook is the abstract Airflow class from which hook classes inherit.
• get_connget_conn is a hook method that establishes or returns a connection to an external system.
• get_connectionget_connection is a BaseHook method that retrieves a stored Airflow connection by connection identifier.
• HashiCorp VaultHashiCorp Vault is a platform for securely storing, managing, and issuing secrets and credentials.
• uriA URI is a standardized string that identifies a resource by name, location, or other identifying information.
• pollingPolling is the technique of repeatedly checking an external system or condition at intervals until a desired state is observed.
• attack surfaceAn attack surface is the collection of possible points through which an attacker could attempt to gain unauthorized access to a system.
• dag.testdag.test is an Airflow DAG method that executes a DAG in a local test context to simulate a DAG run.
• context managerA Python context manager is a class-based language feature that acquires resources in __enter__() and releases them in __exit__().
• dag_idA dag_id is the identifier assigned to an Airflow DAG.
• left-shift operationThe left shift operator is a Python operator that shifts the binary representation of an integer to the left and is overloaded by Airflow for task dependency notation.
• max active runsmax_active_runs_per_dag is an Airflow DAG setting that limits the number of simultaneously active runs of a DAG.
• values.yaml fileA Helm values file is a configuration file containing values that parameterize a Helm chart deployment.
• Git synchronizationGit synchronization is the process of automatically copying the current contents of a Git repository into another environment.
• persistent volumeA PersistentVolume is Kubernetes storage provisioned for use by workloads independently of a pod's lifecycle.
• WeaviateWeaviate is an open-source vector database that stores objects and their vector representations for semantic search and machine-learning applications.
• task_idA task_id is the primary identifier assigned to an Airflow task within a DAG.
• factoryA factory function is a function that accepts configuration and constructs recurring workflow structures such as tasks or DAGs.
• symmetric key encryptionSymmetric encryption is encryption that uses the same secret key for encryption and decryption.
• securitySecurity is the protection of systems and data against unauthorized access, misuse, alteration, or disruption.
• generatorA generator is a Python iterable that produces values incrementally rather than materializing them all at once.
• MessageQueueTriggerMessageQueueTrigger is an Airflow trigger implementation that asynchronously monitors a message queue for events matching its conditions.
• DockerDocker is a platform for building, distributing, and running applications in containers.
• role-based access controlRole-based access control is an authorization technique that grants permissions according to assigned user or group roles.
• frequency-based scheduleA time-based schedule triggers workflow executions at predetermined times or regular temporal intervals.
• persistent volume claimA persistent volume claim is a Kubernetes resource request for persistent storage with specified capacity and access requirements.
• Airflow deploymentAn Airflow instance is an independently configured deployment of the Airflow platform with its own components and operational environment.
• PandasOperatorPandasOperator is a custom Airflow operator that executes reusable pandas data transformations with configurable input and output callables.
• batchingA batch-oriented data pipeline processes data in discrete scheduled or incrementally scheduled groups rather than continuously.
• directory serviceA directory service is a storage system that manages information about resources such as users and services and supports querying that information.
• collectionA collection is a named group of related objects with defined properties and vectorization settings in Weaviate.
• S3CopyObjectOperatorS3CopyObjectOperator is an Airflow operator that copies an object from one Amazon S3 bucket or location to another.
• docker rundocker run is a Docker CLI command that creates and starts a container from an image.
• triggering asset eventstriggering_asset_events is an Airflow context value containing the asset events that triggered a consumer DAG run, grouped by asset.
• EmptyOperatorEmptyOperator is an Airflow operator that represents a task with no substantive work.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.