liveBook · Manning

Data quality

As explained in Effective Data Analysis

Data quality is the degree to which data is accurate, complete, consistent, timely, and fit for its intended analytical purpose.

Data quality is the degree to which data is accurate, complete, consistent, timely, and usable for its intended analysis. Reliable data supports decisions; a large dataset or a well-stocked warehouse does not, by itself, guarantee reliable findings. section 11.3.2 section 11.1

Check four dimensions against the data’s intended use:

  • Accuracy: Do records represent the real-world process, without erroneous values or duplicates? section 11.3.2
  • Completeness: Are required fields present, such as user IDs and timestamps in page views? section 11.3.2
  • Consistency: Do values follow agreed rules, including a consistent treatment of time zones across sources? section 11.3.2
  • Timeliness: Is the data updated and available when a decision needs it? section 11.3.2

Before analyzing a dataset, ask who checks for duplicate, missing, or inaccurate records and what those checks do. Also establish each row’s grain and whether the data was pre-aggregated: aggregation can discard information needed for a question. The right checks depend on the organization and task, so confirm them with the team maintaining the warehouse rather than assuming ingestion is error-free. section 6.2.3

Tests tied to expected values

For daily forecast data, tests can require unique dates, humidity between 0 and 100, nonnegative wind speed, and no missing values. Each assertion turns an expectation into a detectable failure; it does not make the source correct on its own. section 11.3.2

Build checks into extraction, loading, and transformation workflows so problems can be found before data reaches analysts. Responses may include correcting records, removing unusable rows, or inferring missing values where appropriate. Keep the analytical purpose in view: duplicate ticket IDs and a bad date can degrade a metric even if the rest of its dataset looks usable. Clean, structured tables make subsequent analysis faster and easier to share. section 11.3.2 section 7.2.1 section 11.1

How Effective Data Analysis covers it

How deep it goes
Defines four quality dimensions and illustrates checks and remedies, while leaving the choice of questions dependent on the organization and task.
Examples
Uses page-view fields, forecast-data assertions, and duplicate ticket IDs with a bad date to make quality problems concrete.
What it assumes
Builds on familiarity with datasets and columns while defining the quality dimensions through concrete examples.
What you can do after
Practice/perform it by questioning warehouse checks and testing for duplicates, missing values, and invalid ranges.
How it connects
Links data quality to data management, ETL, data testing, analytics engineering, and metric design.

Also explained in

Compare how 10 books explain data quality →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Effective Data Analysis. The section links point to the parts of the book it draws on.