Apache Airflow schedules and orchestrates data workflows. In credit scoring, a monthly workflow can merge usage, bureau, and demographic data into a credit data mart. section 5.1.3
Production financial pipelines may add controls beyond a minimal workflow:
- Track lineage, resource usage, revisions, and audit activity. section 5.1.3
- Retry failed data tasks and mark unresolved bureau data as partial coverage. section 5.1.3
- Coordinate daily or hourly incremental merges with heavier monthly aggregations, and apply encryption and role-based access controls. section 5.1.3
Listing 5.2 A minimal Airflow DAG for BFSI monthly merges
from airflow import DAG
from airflow.providers.postgres.operators.postgres import PostgresOperator
from datetime import datetime
default_args = {
"owner": "credit_pipeline",
"start_date": datetime(2024, 1, 1),
"depends_on_past": False,
"retries": 1
}
with DAG(
dag_id="credit_pipeline_dag",
default_args=default_args,
schedule="@monthly", #1
catchup=False
) as dag:
monthly_usage = PostgresOperator(
task_id="create_monthly_usage",
sql="sql/create_monthly_usage.sql", #2
postgres_conn_id="bfsidb"
)
bureau_lookup = PostgresOperator(
task_id="create_bureau_lookup",
sql="sql/create_bureau_lookup.sql",
postgres_conn_id="bfsidb"
)
demographics = PostgresOperator(
task_id="create_demographics",
sql="sql/create_demographics.sql",
postgres_conn_id="bfsidb"
)
credit_data_mart = PostgresOperator(
task_id="create_credit_data_mart",
sql="sql/credit_data_mart.sql", #3
postgres_conn_id="bfsidb"
)
monthly_usage >> bureau_lookup >> demographics >> credit_data_mart
A minimal monthly DAG defines PostgreSQL tasks for creating the data sources used in a credit-data merge. section 5.1.3
