A missing value marks data that is absent or unknown. Treat it as a data-quality signal: detect where it occurs, decide whether the field is required, and choose a cleanup policy before analysis or downstream processing. section 6.1 section 6.2
A practical missing-value workflow can:
- Count nulls by column with pandas using df.isnull().sum(), then inspect which fields contain incomplete records. section 6.1
- Remove columns that are primarily null when their contents provide little usable information; the demonstrated threshold removes columns exceeding 50% null values. section 6.2
- Include missing-value expectations in an AI prompt when rules vary by dataset, while keeping the resulting cleanup structured and programmable. section 6.0 section 6.2
Detecting and reducing null-heavy data
A pandas DataFrame contains one missing age and one missing purchase amount. Running df.isnull().sum() reports one null in each column. In a separate cleanup example, optional_note is null for every row, so filtering columns whose null count exceeds half the row count removes optional_note while retaining a row whose preferred_store_location is missing. section 6.1 section 6.2
Listing 6.3 Python - Fixing Data Quality Issues
import pandas as pd
# Create a sample DataFrame
df = pd.DataFrame({
'customer_id': [1, 2, 2, 3, 3], #A
'preferred_store_location': ['NY', 'CA', 'CA', 'TX', None], #B
'purchase_amount': [200, 300, 300, 400, 400], #C
'optional_note': [None, None, None, None, None] #D
})
# Remove duplicate rows
df = df.drop_duplicates() #E
# Remove columns that are primarily null (e.g., more than 50% null values)
threshold = 0.5 * len(df) #F
df = df.loc[:, df.isnull().sum() <= threshold] #G
# Output the cleaned DataFrame
print(df) #H
Inspect the complete pandas cleanup pattern for duplicate removal, null-column thresholding, and cleaned-table output. section 6.2






































