liveBook · Manning

Missing value

As explained in Learn AI Data Engineering

A null value is a marker representing the absence or unknown status of a value in a data field.

A missing value marks data that is absent or unknown. Treat it as a data-quality signal: detect where it occurs, decide whether the field is required, and choose a cleanup policy before analysis or downstream processing. section 6.1 section 6.2

A practical missing-value workflow can:

  • Count nulls by column with pandas using df.isnull().sum(), then inspect which fields contain incomplete records. section 6.1
  • Remove columns that are primarily null when their contents provide little usable information; the demonstrated threshold removes columns exceeding 50% null values. section 6.2
  • Include missing-value expectations in an AI prompt when rules vary by dataset, while keeping the resulting cleanup structured and programmable. section 6.0 section 6.2

Detecting and reducing null-heavy data

A pandas DataFrame contains one missing age and one missing purchase amount. Running df.isnull().sum() reports one null in each column. In a separate cleanup example, optional_note is null for every row, so filtering columns whose null count exceeds half the row count removes optional_note while retaining a row whose preferred_store_location is missing. section 6.1 section 6.2

Listing 6.3 Python - Fixing Data Quality Issues
import pandas as pd
 
# Create a sample DataFrame
df = pd.DataFrame({
    'customer_id': [1, 2, 2, 3, 3],  #A
    'preferred_store_location': ['NY', 'CA', 'CA', 'TX', None],  #B
    'purchase_amount': [200, 300, 300, 400, 400],  #C
    'optional_note': [None, None, None, None, None]  #D
})
 
# Remove duplicate rows
df = df.drop_duplicates()  #E
 
# Remove columns that are primarily null (e.g., more than 50% null values)
threshold = 0.5 * len(df)  #F
df = df.loc[:, df.isnull().sum() <= threshold]  #G
 
# Output the cleaned DataFrame
print(df)  #H

Inspect the complete pandas cleanup pattern for duplicate removal, null-column thresholding, and cleaned-table output. section 6.2

How Learn AI Data Engineering covers it

How deep it goes
Demonstrates null counting and null-heavy column removal, but does not specify value imputation policies.
Examples
Uses missing age, purchase amount, store location, and an entirely null optional-note column as concrete examples.
What it assumes
Assumes basic dataset handling for traditional pandas checks while allowing AI prompts to express rules without pandas expertise.
What you can do after
Diagnose/debug with pandas by locating nulls and filtering columns whose missing-value proportion exceeds a chosen threshold.
How it connects
Connects missing values with data quality, pandas, duplicate removal, format validation, and AI-assisted governance rules.

Also explained in

Compare how 39 books explain missing value →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.