A missing value is an absent observation where a value might be expected. It can mean “not applicable,” as when eldest-child age is absent for a person with no children, or “unknown,” as when a true age was not collected. section 9.1
Missingness can itself carry information. Rare nulls may form useful outliers, and nonrandom failures—such as temperature sensors failing at extreme temperatures—can reveal conditions associated with the absence. Filling the value should therefore preserve an indicator that it was missing. section 9.1
A practical treatment separates interpretation from imputation:
- Identify whether the absence is legitimate or a data artifact caused by inaccessible information, joins, appends, or processing errors. section 9.1
- Record missingness before filling values, using per-column indicators or a row-level count of missing fields. section 9.1
- Fill numeric values with a median and categorical values with a mode when a detector requires complete input; treat these as preprocessing choices rather than recovered truth. section 9.0 section 9.1
- Separate likely data artifacts before identifying statistical outliers in the remaining data, because missingness and unusual values may represent different problems. section 9.1
A conditional absence
In a table containing Number of Children and Age of Eldest Child, a zero child count with a missing eldest-child age is expected. A positive child count with the same missing age indicates an unknown value or collection problem and deserves separate attention. section 9.1
Listing 9.1 Filling None and NaN values
import pandas as pd
import numpy as np
df = pd.DataFrame({"A": [1, 2, np.NaN, 4, 5, None],
"B": ['A', 'B', None, "B", "B", "F"]})
df_filled = df.copy()
df_filled['A Null'] = df['A'].isna()
df_filled['B Null'] = df['B'].isna()
df_filled['Num Null'] = df.isna().sum(axis=1)
df_filled['A'] = df['A'].fillna(df['A'].median())
df_filled['B'] = df['B'].fillna(df['B'].mode()[0])
print(df_filled)
Shows how to preserve null flags, count missing fields, and fill numeric and categorical columns with median and mode values. section 9.1
