liveBook · Manning

Sparsity

As explained in Outlier Detection in Python

Data sparsity is a condition in which observed records occupy only a very small portion of the possible data space.

Data sparsity occurs when a dataset occupies only a minute fraction of a very large possible data space, leaving most combinations unobserved. section 3.1.1

Twenty categorical features

With 10,000 rows and 20 categorical columns, each containing four equally common values, one specific combination has probability 0.25^20. Under independence, that combination is extraordinarily unlikely to appear, so the dataset would need to be many orders of magnitude larger to observe it even once. Real data can be less sparse when features correlate or column frequencies are skewed, although many features may still overwhelm the available records. section 3.1.1

How Outlier Detection in Python covers it

How deep it goes
Defines sparsity and works through joint probabilities, stopping short of presenting a detection algorithm.
Examples
Uses 10,000 rows, 20 four-valued categorical columns, and the probability 0.25^20 to illustrate unobserved combinations.
What it assumes
Assumes familiarity with records, categorical features, marginal probabilities, joint probabilities, and independence.
How it connects
Connects sparsity with categorical features, marginal probabilities, independence, correlation, and skewed distributions.

Also explained in

Compare how 6 books explain sparsity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Outlier Detection in Python. The section links point to the parts of the book it draws on.