liveBook · Manning

Sum rule

As explained in Math and Architectures of Deep Learning

The probability rule stating that the total probability over all possible outcomes equals one.

The sum rule says that probabilities assigned to mutually exclusive events add to the probability that one of those events occurs. When the events are exhaustive, covering every possible outcome, their total probability is 1. It provides the normalization condition for probability distributions. section 5.10

For a discrete variable, add the probabilities of all possible values. For a continuous variable, replace the sum with an integral: the area under a probability density curve over its full domain equals 1. For a multidimensional variable, the corresponding volume under the density hypersurface also equals 1. section 5.6

The probability density function p(x) for a continuous random variable X is defined as the probability that X lies between x and x + δx with δx → 0

p(x) = limδx→0 probability(x ≤ X < x + δx)

Relates probability density at x to the probability of X falling in an infinitesimally small interval around x. section 5.6

Summing a joint probability table

A joint table for weight categories W and height categories H contains probabilities such as p(E2, F3) = 0.04. Adding every cell gives 1 because one weight–height combination must occur. Adding a row or column instead produces a marginal probability, and all marginal probabilities also sum to 1. section 6.1.1

Table 6.1 Example population sizes and joint probability distribution for variables W = {E1, E2, E3} and H = {F1, F2, F3} weights and heights of adult Statsville residents), showing marginal probabilities

Less than 60 kg (E1)

Between 60 and 90 kg (E2)

More than 90 kg (E3)

Marginals for Fs

Less than 160 cm (F1)

pop. = 20,000

p(E1, F1) = 0.2

pop. = 4,000

p(E2, F1) = 0.04

pop. = 2,000

p(E3, F1) = 0.02

pop. = 26,000;

p(F1) = 0.2 + 0.04 + 0.02 = 0.26

Between 160 cm and 183 cm (F2)

pop. = 4,000

p(E1, F2) = 0.04

pop. = 40,000

p(E2, F2) = 0.4

pop. = 4,000

p(E3, F2) = 0.04

pop. = 48,000;

p(F2) = 0.04 + 0.4 + 0.04 = 0.48

More than 183 cm (F3)

pop. = 2,000

p(E1, F3) = 0.02

pop. = 4,000

p(E2, F3) = 0.04

pop. = 20,000

p(E3, F3) = 0.2

pop. = 26,000;

p(F3) = 0.02 + 0.04 + 0.2 = 0.26

Marginals for Es

p(E1)

= 0.2 + 0.04 + 0.02 = 0.26

p(E2)

= 0.04 + 0.4 + 0.04 = 0.48

p(E3)

= 0.02 + 0.04 + 0.2 = 0.26

Total pop. = 100,000;

Total prob = 1

Displays joint probabilities for weight and height categories together with their row and column marginals. section 6.1.1

How Math and Architectures of Deep Learning covers it

How deep it goes
Defines the rule, explains exhaustive and mutually exclusive events, and derives its integral form without implementation detail.
Examples
Uses continuous density areas, multidimensional density volumes, and a weight–height joint probability table.
What it assumes
Assumes basic probability notation and integrals while introducing discrete and continuous random variables from their possible outcomes.
What you can do after
Evaluate/understand whether discrete and continuous probability assignments cover all possible outcomes and normalize correctly.
How it connects
Connects probability distributions with random variables, probability density functions, joint probability, marginal probability, and independence.
Cover of Math and Architectures of Deep Learning
liveBookmanning.com
Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.