Before using an analysis to make a business decision, ask whether the data describes the activity you care about. A spreadsheet can calculate a trend from two points, but that does not make the result meaningful. Data quality is about both what records say and whether their structure supports reliable analysis. Use the best data available, and make any compromises consciously. section 2.0 section 2.1
Investigate quality systematically rather than looking for a single automated fix:
- Inspect sample records, dataset size, column information, summary statistics, missing values, and value counts. Ask a generative AI assistant for suitable pandas checks, then run and inspect the code yourself. section 2.3
- Check whether data is complete and current, whether duplicates exist, and whether values agree across sources. Check that relationships between fields remain meaningful. section 2.2
- Clarify what each variable means in the business context. If a surprising result remains unexplained, ask the data provider rather than treating an AI explanation as proof. section 2.1 section 2.7
Which field identifies a customer?
Suppose an order dataset has both customer_id and customer_unique_id. Ask an AI assistant how to compare their distinct counts and group one by the other; run the suggested pandas code and examine its output. If customer_id appears to identify records rather than people, using it to count returning customers could mislead. Confirm the fields’ meanings—and an unexpectedly high share of single-order customers—with the relevant team before calling the pattern a quality defect. section 2.3
Unique customer_id values: 99441 Unique customer_unique_id values: 96096 Summary of the relationship between customer_unique_id and customer_id: Min: 1 Max: 17 Mean: 1.0348089410589412 Count of unique customer_id values per customer_unique_id: 1 93099 2 2745 3 203 4 30 5 8 6 6 7 3 9 1 17 1 Name: customer_id, dtype: int64
The inspection output reports 99,441 distinct customer_id values and 96,096 distinct customer_unique_id values, then shows how many customer_id values belong to each customer_unique_id. This makes the suspected identifier relationship easier to inspect; it does not establish the business meaning of either field. section 2.3
Keep the quality investigation distinct from in-depth analysis, but expect to revisit cleaning and exploratory data analysis. Text reviews need additional care: biased or unrepresentative submissions can distort aggregates, personal information may need anonymization, and domain-specific stopwords can matter. Automated monitoring can flag issues, while periodic expert review can catch nuances it misses. section 2.7 section 5.2.1
