A credit application may lack an employment status, while a customer with no prior credit history may have no credit utilization ratio. A missing value is an absent field for a particular record. Its absence can be a data-quality problem, but it can also carry information about default risk. The goal is to make that distinction before cleaning the data. section 4.4.4 section 5.3.2
Use a deliberate preprocessing sequence:
- Inspect each column’s missing count and percentage alongside its type, unique values, and basic statistics. Investigate fields with substantial missingness before choosing a treatment. section 5.3.2
- Identify placeholder codes such as −9999 or 8888 separately from null values. Confirm their meanings with the data provider; treating them as ordinary numbers can skew distributions. section 4.4.4
- Choose a treatment that fits the pattern: delete rows when only a small fraction are missing, impute numeric fields, or give missing values their own bin. A baseline workflow drops columns more than 80% empty and fills remaining numeric gaps with column means. section 4.4.4 section 5.3.2
Listing 5.5 Summary function
def summary(df): #1
summ = pd.DataFrame(df.dtypes, columns=['data type'])
summ['#missing'] = df.isnull().sum().values
summ['%missing'] = (df.isnull().sum().values / len(df)) * 100
summ['#unique'] = df.nunique().values
desc = df.describe(include='all').transpose() #2
if 'min' in desc.columns:
summ['min'] = desc['min'].values
else:
summ['min'] = np.nan
if 'max' in desc.columns:
summ['max'] = desc['max'].values
else:
summ['max'] = np.nan
if len(df) > 0:
summ['first value'] = df.iloc[0].values
if len(df) > 1:
summ['second value'] = df.iloc[1].values
if len(df) > 2:
summ['third value'] = df.iloc[2].values
return summ #3
print("Initial data shape:", df.shape) #4
summary_df = summary(df) #5
display(summary_df.head(10)) #6
The pandas `summary(df)` function calculates `#missing` and `%missing` for every column, providing an inspection step before cleaning. section 5.3.2
When an average hides a signal
A missing credit utilization ratio can indicate a customer with no prior credit history. Filling that field with the column mean keeps the training row, but makes the absence less visible and reduces measured variability. A distinct missing category or sentinel may better preserve that signal for a tree-based model; its meaning and risk effect still need checking. section 5.3.2 section 4.4.4
Document why each field was deleted, imputed, or assigned a missing category. Check whether missing and sentinel-coded groups have different default rates, and be ready to justify assumptions that absent data are neutral or average. These choices affect model stability, interpretability, and fair-lending review. section 4.4.4 section 4.6
