Data quality
Data quality is the degree to which data is accurate, complete, consistent, relevant, and suitable for its intended use.
Starting Data Analytics with Generative AI and Python
A data quality issue is a defect in data's accuracy, completeness, consistency, validity, or interpretation.
- How it is used
- Supports decisions by checking whether available data describes the business activity rather than introducing misleading analytical results.
- How deep it goes
- Develops a systematic inspection and validation practice, while rejecting a fully automated, one-size-fits-all quality fix.
- Examples
- Uses pandas inspection and competing customer identifiers to test whether an apparent customer pattern reflects the data’s meaning.
- What you can do after
- Practice/perform structured data checks, investigate suspicious results with providers, and weigh quality compromises before relying on conclusions.
Read how Starting Data Analytics with Generative AI and Python explains data quality →
Effective Data Analysis
Data quality is the degree to which data is accurate, complete, consistent, timely, and fit for its intended analytical purpose.
- How it is used
- Supports reliable analysis and decisions by checking whether data can serve its intended purpose.
- How deep it goes
- Defines four quality dimensions and illustrates checks and remedies, while leaving the choice of questions dependent on the organization and task.
- Examples
- Uses page-view fields, forecast-data assertions, and duplicate ticket IDs with a bad date to make quality problems concrete.
- What you can do after
- Practice/perform it by questioning warehouse checks and testing for duplicates, missing values, and invalid ranges.
Machine Learning System Design
Data quality is the degree to which data is accurate, complete, valid, consistent, timely, and suitable for its intended purpose.
- How it is used
- Protects training and production predictions by checking inputs before aggregation and monitoring their integrity after deployment.
- How deep it goes
- Details preprocessing, missing-data baselines, alarm thresholds, and incident response, but does not specify a complete quality-check implementation.
- Examples
- Uses retail transactions, stock and promotion records, incomplete competitor prices, and missing-feature alarms to make checks concrete.
- What you can do after
- Diagnose/debug with it by checking data defects before retraining and choosing investigation or a fallback when inputs are unreliable.
Read how Machine Learning System Design explains data quality →
Data Without Labels
Data quality is the degree to which data is complete, valid, accurate, representative, available, consistent, timely, and internally coherent for a particular use case.
- How it is used
- Positions data quality as a gate between identifying a business problem and preparing data for modeling.
- How deep it goes
- Defines core quality attributes and gives task-specific checks, but defers detailed quality inspection to a later preparation step.
- Examples
- Uses daily rainfall measurements, annual sales completeness, date-range validity, and stakeholder-verified sales and customer indicators.
- What you can do after
- Evaluate/understand whether a dataset fits a use case and which quality checks must precede preparation and modeling.
The Art of AI Product Development
Data quality is the degree to which data is accurate, complete, consistent, timely, and fit for its intended purpose.
- How it is used
- Supports training-data preparation and language-model output reliability for AI products.
- How deep it goes
- Explains annotation guidelines, agreement checks, class balance, bias, noise, privacy, and knowledge-cutoff risks.
- Examples
- Uses fashion recommendations, labeled classifier examples, biased job associations, and content-generation fact checking.
- What you can do after
- Diagnose training-data and generated-content risks, then apply labeling checks, human review, validation, and fresher data sources.
Read how The Art of AI Product Development explains data quality →
Snowflake Data Engineering
Data quality is the degree to which data is accurate, complete, consistent, valid, and fit for its intended purpose.
- How it is used
- Uses validation, testing, logging, and alerting to keep scheduled data pipelines reliable and fit for downstream consumption.
- How deep it goes
- Explains synchronous and asynchronous tests, violation responses, metric functions, event logging, and alerts without detailing individual quality-rule implementations.
- Examples
- Demonstrates primary-key validation, logging-table test results, scheduled metric functions, event tables, alerts, and stored-procedure pipeline loads.
- What you can do after
- Apply it via Snowflake tasks, data metric functions, logging tables, and alerts to detect violations and trigger operational responses.
Delivering Machine Learning Projects: From design to deployment
Data quality is the degree to which data is accurate, complete, consistent, relevant, timely, and fit for its intended use.
- How it is used
- Uses data quality to locate risks during an early ML data survey before modeling and application delivery.
- How deep it goes
- Inspects records, missingness, placeholders, and visible patterns, while deferring deeper investigation to exploratory data analysis.
- Examples
- Uses smart-building incident text, SQL record samples, missing-value checks, and image collections containing potentially irrelevant content.
- What you can do after
- Evaluate whether data contains actionable information and decide what to investigate, resolve, or avoid.
Read how Delivering Machine Learning Projects: From design to deployment explains data quality →
Architecting an Apache Iceberg Lakehouse
Data quality is the degree to which data is accurate, consistent, complete, timely, and fit for its intended use.
- How it is used
- Guides stakeholder discovery, ingestion design, and consumption-layer architecture by connecting data quality with trust, freshness, and usability.
- How deep it goes
- Connects quality to architectural tradeoffs and shared consumption, but stops short of defining validation rules or quality metrics.
- Examples
- Uses reports, models, dashboards, BI tools, notebooks, ML platforms, and replicated departmental datasets as examples.
- What you can do after
- Evaluate architecture choices by relating ingestion and consumption patterns to data trust, consistency, latency, and duplication.
Read how Architecting an Apache Iceberg Lakehouse explains data quality →
Machine Learning Platform Engineering
Data quality is the degree to which data is accurate, complete, consistent, timely, valid, and fit for its intended purpose.
- How it is used
- Runs quality assurance before model training to prevent defective structured data from entering production ML workflows.
- How deep it goes
- Defines schema and type checks with runnable validation, while deferring extensive exploration and deeper defect analysis.
- Examples
- Uses a MinIO-hosted Parquet dataset and checks for user_id, item_id, and rating columns before training.
- What you can do after
- Diagnose production input defects with schema and data-type checks before launching training runs.
Read how Machine Learning Platform Engineering explains data quality →
Financial AI in Practice
Data quality is the degree to which data is accurate, complete, consistent, timely, and fit for its intended purpose.
- How it is used
- Protects financial AI and risk decisions by ensuring inputs are accurate, timely, complete, and governed.
- How deep it goes
- Explains failure consequences and pipeline controls, but gives no explicit data-quality measurement procedure.
- Examples
- Uses noisy or outdated LLM inputs, credit data feeds, and real-time loan-application features as examples.
- What you can do after
- Operate a data asset layer that governs feeds, transformations, privacy constraints, and task-specific stores.
Related concepts
- completenesshas part · Completeness is the extent to which all elements required or relevant to a defined scope are present, represented, or accounted for.
- accuracyhas part · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.
- validityhas part · Validity is the degree to which data, measures, outputs, or evaluations satisfy the requirements and accurately represent the intended use or construct.
- hyperparameter tuningcontrasts with · Hyperparameter tuning is the process of selecting or adjusting configurable model or procedure settings and evaluating them against a chosen objective.
- consistencycontrasts with · Consistency is the property of related data, behavior, outputs, or states agreeing with one another or conforming to defined rules across relevant instances or operations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.