Data quality is the fitness of a dataset for a particular business problem. Dirty, incorrect, or irrelevant data can undermine an analysis before modeling begins: good-quality outputs depend on good-quality inputs. section 1.2.3
Assess quality against the intended use, not just the dataset in isolation:
- Check completeness: are important observations or variables missing? A year of monthly sales data should include all 12 months. section 1.2.3
- Check validity: do observations match the use case’s scope? Data from 2014 does not fit a study restricted to 2015–2019. section 1.2.3
- Check accuracy: are measurements and calculated business figures correct? Inaccurate data leads to inaccurate insights and faulty actions. section 1.2.3
- Also assess representativeness, availability, consistency, timeliness, and integrity. section 1.7
Rainfall prediction
To predict rainfall at a location from last year’s daily measurements, seek few missing days, accurate measurements, coverage of the relevant local area, and data you can access and use. Measurements from a distant area or a dataset full of gaps may be a poor fit even if it contains rainfall figures. section 1.2.3
During data discovery, obtain data relevant to the business problem and calculate basic indicators such as total sales, customer counts, monthly trends, and discounts. Have business stakeholders verify those figures; if they are wrong, refine the business rules. Then inspect quality issues such as missing values and outliers, look for patterns and relationships, and proceed to cleaning, preprocessing, and preparation. section 11.3
