liveBook · Manning

Missing not at random

As explained in Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

Missing not at random describes a missingness mechanism in which missingness depends on unobserved values or other unaccounted-for information.

Missing not at random describes missingness that depends on the unobserved value itself or other unavailable information. It needs specialized treatment. When the mechanism is understood, adding proxy features can make the problem more like missing at random and support ordinary imputation. section 6.1.1

Treat MNAR as a data-generation problem before choosing an algorithm’s default missing-value behavior:

  • Investigate why values are missing and distinguish MNAR from missing completely at random or missing at random. section 7.1.5
  • Avoid relying automatically on XGBoost or LightGBM missing-value handling, which may mimic an average case and underperform. section 6.1.2
  • Add missingness indicators or impute using other features when the missingness pattern carries useful information. section 6.1.2

Accommodation reviews

In accommodation data, missing review-related values corresponded to properties with no reviews or too few reviews to compute a score. Because missingness represented a legitimate business condition, replacing values with -1—below the observed distributions—helped models recognize the absence of reviews. section 7.1.5

X.isna().sum()

Check missing-value counts with X.isna().sum() before investigating the mechanism behind the missingness. section 7.1.5

How Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI covers it

How deep it goes
Defines MNAR and gives practical responses, but defers specialized treatment beyond proxy-feature collection and alternative imputation.
Examples
Uses accommodation reviews, missingness counts, and negative boundary-value replacement as concrete diagnostics and treatments.
What it assumes
Assumes familiarity with missing data and machine-learning features while distinguishing MCAR, MAR, and MNAR explicitly.
What you can do after
Diagnose/debug with it by investigating missingness causes and selecting indicators, proxy features, or informed imputation.
How it connects
Connects MNAR to missing completely at random, missing at random, imputation, and gradient-boosted decision trees.
Cover of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
liveBookmanning.com
Read Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI. The section links point to the parts of the book it draws on.