Data testing
Data testing is the systematic verification of data against expected correctness, completeness, consistency, validity, and other quality requirements.
Delivering Machine Learning Projects: From design to deployment
Data testing is the automated checking of data values, distributions, relationships, and construction rules against expected conditions.
- How it is used
- Checks data flowing through development and production pipelines so model work relies on data with expected properties.
- How deep it goes
- Provides test categories, concrete checks, and cost guidance, but does not specify a testing framework or full implementation.
- Examples
- Uses duplicate records, arrival volumes, missing columns, sensor-related nulls, distribution bounds, and incorrectly constructed features.
- What you can do after
- Build it from scratch by turning survey findings and pipeline expectations into checks that continue into production.
Read how Delivering Machine Learning Projects: From design to deployment explains data testing →
Snowflake Data Engineering
Data testing is the practice of systematically checking data against expected correctness, completeness, consistency, validity, and other quality conditions.
- How it is used
- Adds quality-validation steps and response actions to data pipelines so analytics consumers can use data confidently.
- How deep it goes
- Develops task-based checks, violation logging, scheduled metric functions, and alerts, while recommending additional tests for thorough coverage.
- Examples
- Uses PARTNER and PRODUCT loads with a null rating and invalid category, recorded in DQ_LOG by dependent tasks.
- What you can do after
- Operate/configure it through pipeline tasks, logging, scheduled data metric functions, and alerts for quality violations.
Related concepts
- data qualitymeasures · Data quality is the degree to which data is accurate, complete, consistent, relevant, and suitable for its intended use.
- quality measuremeasures · A quality measure is a criterion or quantitative indicator used to assess one or more quality attributes of data, products, processes, or outcomes.
- pipelinepart of · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
- data metric functionimplemented by · A data metric function is a Snowflake function that evaluates a data quality metric for an assigned table or view on a defined schedule.
- data integritymeasures · Data integrity is the property that data remains accurate, complete, consistent, and valid, with changes made only through authorized actions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.