liveBook · Manning

Missing data indicator

As explained in Starting Data Analytics with Generative AI and Python

A missing value indicator is a boolean representation marking whether each data value is missing.

When missing values appear together, the pattern may reveal a data-quality problem worth investigating. A missing data indicator turns each value into a simple boolean: True when missing and False otherwise. section 2.5.2

In pandas, df.isna() creates an indicator DataFrame with the same shape as the original. Correlating its columns shows whether missing values tend to co-occur; a heatmap makes those relationships easier to inspect. Treat generated code as a starting point: execute it, inspect the outputs, and check suspicious patterns against the raw records. section 2.5.2

A practical pattern check

For columns A, B, C, and D, create missing_indicator = df.isna(), compute missing_indicator.corr(), and display the result with a Seaborn heatmap. Strong positive values suggest that missingness in two columns often occurs in the same rows. section 2.5.2

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# Sample DataFrame with missing values
data = {'A': [1, None, 3, 4, None],
        'B': [1, 2, None, 4, None],
        'C': [None, 2, 3, None, 5],
        'D': [1, 2, 3, 4, 5]}

vdf = pd.DataFrame(data)

# Create a DataFrame indicating which values are missing
missing_indicator = df.isna()

# Compute the correlation between the missing value indicators
missing_corr = missing_indicator.corr()

# Visualize the correlation matrix using a heatmap
sns.heatmap(missing_corr, annot=True, cmap='coolwarm', vmin=-1, vmax=1)
plt.title("Correlation of Missing Value Indicators")
plt.show()

Use the executable listing to inspect the indicator creation, correlation calculation, and heatmap workflow. section 2.5.2

How Starting Data Analytics with Generative AI and Python covers it

How deep it goes
Defines boolean indicators, computes pairwise correlations, and visualizes them in a heatmap without prescribing a remedy.
Examples
Uses a sample DataFrame with columns A through D and a heatmap showing correlations between missing values.
What it assumes
Assumes a pandas DataFrame and introduces boolean missingness indicators through df.isna().
What you can do after
Diagnose/debug with it by identifying columns whose missing values tend to occur together.
How it connects
Connects missing values with pandas, correlation matrices, heatmaps, and Seaborn visualization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Starting Data Analytics with Generative AI and Python. The section links point to the parts of the book it draws on.