A directed acyclic graph (DAG) represents tasks and their dependencies without directed cycles. In a credit-data pipeline, an Airflow DAG can coordinate monthly usage, bureau, and demographic data merges into a final credit data mart. section 5.1.3
Listing 5.2 A minimal Airflow DAG for BFSI monthly merges
from airflow import DAG
from airflow.providers.postgres.operators.postgres import PostgresOperator
from datetime import datetime
default_args = {
"owner": "credit_pipeline",
"start_date": datetime(2024, 1, 1),
"depends_on_past": False,
"retries": 1
}
with DAG(
dag_id="credit_pipeline_dag",
default_args=default_args,
schedule="@monthly", #1
catchup=False
) as dag:
monthly_usage = PostgresOperator(
task_id="create_monthly_usage",
sql="sql/create_monthly_usage.sql", #2
postgres_conn_id="bfsidb"
)
bureau_lookup = PostgresOperator(
task_id="create_bureau_lookup",
sql="sql/create_bureau_lookup.sql",
postgres_conn_id="bfsidb"
)
demographics = PostgresOperator(
task_id="create_demographics",
sql="sql/create_demographics.sql",
postgres_conn_id="bfsidb"
)
credit_data_mart = PostgresOperator(
task_id="create_credit_data_mart",
sql="sql/credit_data_mart.sql", #3
postgres_conn_id="bfsidb"
)
monthly_usage >> bureau_lookup >> demographics >> credit_data_mart
Inspect a minimal Airflow DAG for monthly BFSI merges, implemented with PostgreSQL operators. section 5.1.3
The minimal workflow is a starting point for production: financial pipelines may add retries and partial-coverage flags, lineage and revision records, security controls, audit logs, and schedules at multiple frequencies. section 5.1.3
