A credit team cannot score an applicant reliably by pointing a model at scattered bureau, usage, and demographic records. A pipeline turns those inputs into a usable credit data mart, then carries prepared data through modeling, validation, scoring, and inference. Each step produces an output needed by a later step; the model is only one part of the workflow. section 5.1.3 section 5.3
The steps depend on the financial job:
- For batch data preparation, schedule dependent ingestion and merge tasks in an Airflow directed acyclic graph (DAG). A nightly personalization pipeline can extract ledger and clickstream data, aggregate behavior over time windows, then join it to financial snapshots in a User 360 mart. section 5.1.3 section 13.1.2
- For model training, make validation part of the pipeline. A fraud-classification pipeline combines oversampling with LightGBM and applies oversampling within each cross-validation fold during hyperparameter search, preventing data leakage. section 8.3
- For financial news, ingest and clean text, create embeddings with metadata, retrieve relevant documents, and pass that context to an LLM. Its structured JSON output can become a qualitative signal for a quantitative engine. section 11.1.4
- For recommendations, process user context and new content separately: recompute user features in batches while ingesting, enriching, and vectorizing content as it arrives. Match their vectors at interaction time without rebuilding every input. section 13.0 section 13.3.1
Listing 5.2 A minimal Airflow DAG for BFSI monthly merges
from airflow import DAG
from airflow.providers.postgres.operators.postgres import PostgresOperator
from datetime import datetime
default_args = {
"owner": "credit_pipeline",
"start_date": datetime(2024, 1, 1),
"depends_on_past": False,
"retries": 1
}
with DAG(
dag_id="credit_pipeline_dag",
default_args=default_args,
schedule="@monthly", #1
catchup=False
) as dag:
monthly_usage = PostgresOperator(
task_id="create_monthly_usage",
sql="sql/create_monthly_usage.sql", #2
postgres_conn_id="bfsidb"
)
bureau_lookup = PostgresOperator(
task_id="create_bureau_lookup",
sql="sql/create_bureau_lookup.sql",
postgres_conn_id="bfsidb"
)
demographics = PostgresOperator(
task_id="create_demographics",
sql="sql/create_demographics.sql",
postgres_conn_id="bfsidb"
)
credit_data_mart = PostgresOperator(
task_id="create_credit_data_mart",
sql="sql/credit_data_mart.sql", #3
postgres_conn_id="bfsidb"
)
monthly_usage >> bureau_lookup >> demographics >> credit_data_mart
The minimal Airflow DAG makes batch orchestration inspectable: scheduled SQL tasks create partial credit datasets before their merge into the data mart. section 5.1.3
A working DAG is not yet a production pipeline. Credit-data runs also need lineage and revision records, retries or explicit partial-coverage tags, appropriate schedules, encryption, access controls, audit logs, and a versioned data dictionary. These controls help prevent missing inputs from silently becoming valid-looking model data and let compliance teams trace changes. Cost-aware ML–LLM routing and human oversight become further design concerns when a pipeline drives proactive customer interventions. section 5.1.3 section 14.0
