DataConsistencyChecker (DCC) is a data-quality and interpretable outlier-detection tool. It discovers common patterns within and across columns, then flags row-level exceptions rather than judging values by magnitude or category frequency alone. section 13.3.4
Its tests include:
- Detecting unusual decimal patterns, such as 43.2233 among mostly differently formatted values. section 13.3.4
- Predicting one column from other row values with simple decision trees or linear regression. section 13.3.4
- Finding aligned zero, nonzero, null, or negative-value patterns and arithmetic relationships such as sums, ratios, products, differences, minima, maxima, and means. section 13.3.4
A typical workflow initializes a pandas DataFrame, runs check_data_quality(), and reviews findings with display_detailed_results() or one result at a time through display_next(). More than 150 simple tests can produce substantial output, so result filters help focus inspection. section 13.3.4
Listing 13.13 Example executing DataConsistencyChecker
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
import sys
sys.path.insert(1, '..')
from check_data_consistency import DataConsistencyChecker
data = fetch_openml('abalone', version=1, parser='auto')
df = pd.DataFrame(data.data, columns=data.feature_names)
dc = DataConsistencyChecker(verbose=0)
dc.init_data(df)
dc.check_data_quality()
dc.display_detailed_results(test_id_list=['LARGER'])
Python example using the abalone dataset, pandas, and a filtered LARGER consistency test. section 13.3.4
