Data quality is the degree to which data is accurate, complete, consistent, valid, and fit for its intended purpose. In production pipelines, it protects data integrity while pipelines handle failures, restarts, unexpected types, inconsistencies, outages, and access violations. section 4.5
Implement data quality as an operational pipeline capability:
- Add synchronous checks inside the pipeline or run ingestion and testing asynchronously. section 14.6
- Choose responses to violations, including stopping execution, notifying recipients, or logging results. section 14.6
- Validate feasible rules such as primary keys, and prevent adverse scenarios from duplicating or losing data. section 4.5
- Use built-in or custom data metric functions for scheduled measurements, and Snowflake alerts for condition-based notifications. section 14.6
Logging violations and alerting operators
Create a data-testing task that inserts rows into a logging table only when quality rules fail, or records successful checks when business users require confirmation. Store scheduled data metric function results in the event table, then configure a Snowflake alert to email designated recipients when violations occur within a chosen interval. section 14.6
use database BAKERY_DB;
use schema TRANSFORM;
create or replace procedure LOAD_CUSTOMER_SUMMARY_ORDERS()
returns varchar
language sql
as
$$
begin
SYSTEM$LOG_DEBUG('LOAD_CUSTOMER_SUMMARY_ORDERS begin ');
truncate table SUMMARY_ORDERS;
insert into SUMMARY_ORDERS... #1
return 'Load completed. ' || SQLROWCOUNT || ' rows inserted.';
exception
when other then
return 'Load failed with error message: ' || SQLERRM;
end;
$$
;
Use the stored procedure scaffold to inspect a modular pipeline load with logging, truncation, insertion, row-count reporting, and exception handling. section 4.5
