Chapter 5: Probability distributions in machine learning

Introduction

This introduction motivates probability distributions as tools for modeling uncertain, high-dimensional machine-learning data and previews the chapter’s probabilistic and PyTorch-based approach.

• Probabilistic modeling of feature clusters: Explain how a probability distribution can model clusters of high-dimensional feature vectors rather than merely produce a class decision. Describe how fitting such distributions can reveal structure or correlations in data, such as overlap between groups relevant to recommendations.
  Why it matters: Probability distributions let machine-learning systems represent and analyze uncertain structure in data, not just output a label.
• Probabilistic machine-learning applications: Describe why some machine-learning tasks use probabilistic outputs, including pixelwise foreground probabilities in semantic segmentation. State that probabilistic models also occur in unsupervised or minimally supervised learning, including variational autoencoders.
  Why it matters: Probabilistic outputs are useful when a simple fixed-class decision is not an adequate description of the task.
• Chapter scope and PyTorch distributions: State that the chapter introduces probability, probability distributions including multivariate distributions, and a geometrical view of multivariate statistics. Describe PyTorch distributions as the statistical package used to accompany the distributions discussed in the chapter.
  Why it matters: Knowing the chapter’s scope helps connect the mathematical distribution concepts to the PyTorch tools used to work with them.

Key terms: feature vectors, high-dimensional space, probability distributions, clusters, correlation, probabilistic models, semantic segmentation, foreground, background, variational autoencoders

5.1 Probability: The classical frequentist view

This section introduces the frequentist definition of probability as the fraction of a population that satisfies an event's specified criteria.

• Frequentist probability ratio: Explain probability in the classical frequentist view as the number of population members satisfying an event divided by the total population size. Describe that an event can specify one criterion, such as height, or multiple criteria, such as a height and weight range.
  Why it matters: This population-counting view provides the starting meaning of probability for the distributions and joint events developed next.
• Probability upper bound: State why a frequentist probability cannot exceed 1: every event population is a subset of the total population, so the favorable-outcome count cannot be larger than the possible-outcome count.
  Why it matters: This bound is a basic check on whether a probability computed from population counts is sensible.

Key terms: frequentist paradigm, event, favorable outcomes, possible outcomes, population, superset, probability

5.1.1 Random variables

This section defines random variables as numerical quantities whose possible values or ranges have probabilities, distinguishing discrete and continuous cases.

• Random variable definition: Explain that a random variable is a numerical entity associated with an event, such as height, weight, a stock index, or infection count, and that each allowed value or value range has an associated probability.
  Why it matters: Random variables specify what quantity a probability statement is actually about.
• Discrete random variables: Describe a discrete random variable as one that takes values from a countable set, using the quantized Statsville weight example in which three weight ranges are encoded as X = 1, 2, or 3 and each value has a fixed probability.
  Why it matters: Discrete variables let categories or quantized numerical ranges be represented as numbered outcomes with probabilities.
• Continuous variables and notation: Describe a continuous random variable as one considered over an infinitesimally small interval such as p(x ≤ X < x + δx) as δx approaches zero, called probability density, and state the uppercase-X and lowercase-x notation convention.
  Why it matters: This distinction and notation prepare you to read continuous distributions and probability-density expressions accurately.

Key terms: random variable, event, allowed value, value range, probability, discrete random variable, countable set, quantized weight, X = 1, X = 2

5.1.2 Population histograms

This subsection shows how a population histogram displays counts for the possible categories of a discrete random variable.

• Population histogram interpretation: Describe how a population histogram represents the possible values of a discrete random variable, with the X-axis showing categories and the Y-axis showing the population count in each category.
  Why it matters: Reading a histogram lets you turn a visual distribution of a population into counts for each possible outcome.
• Exhaustive weight-range counts: Explain how the three Statsville weight ranges S1, S2, and S3 account for the entire adult population, and verify this by summing their counts to 100,000.
  Why it matters: Checking that all category counts add to the total population confirms that every population member has been included.

Key terms: population histogram, discrete random variable, X-axis, Y-axis, S1, S2, S3, entire adult population, population size

5.2 Probability distributions

This section defines a discrete probability distribution using quantized adult weights and explains why its events must be exhaustive and mutually exclusive.

• Discrete probability distribution: Explain that a probability distribution is a mathematical function that takes a random variable value as input and returns the probability of that allowed value, and that it must be defined for every possible value of the variable.
  Why it matters: Probability distributions turn observed populations into a mathematical description of how likely each possible outcome is.
• Exhaustive mutually exclusive ranges: Describe why the weight ranges S1, S2, and S3 form valid distribution events: together they cover every possible value of X, and no single observation can belong to more than one range. State that union denotes coverage across sets and intersection denotes shared members, which is null for every pair of these ranges.
  Why it matters: Exhaustive, nonoverlapping events ensure that each possible outcome receives one unambiguous probability.

Key terms: probability distribution, random variable, allowed values, discrete random variable, quantized weight, exhaustive, mutually exclusive, set union, set intersection, S1

5.3 Basic concepts of probability theory

5.3.1 Probabilities of impossible and certain events

This subsection distinguishes impossible and certain events from merely improbable and highly probable events using probability values.

• Probability endpoint meanings: Explain that an impossible event has probability 0 and an event that occurs with certitude has probability 1, using the sun rising in the west and east as examples.
  Why it matters: Probability values at the endpoints have definite meanings, unlike values that are only close to them.
• Near-endpoint probability values: Distinguish low but nonzero probabilities from zero, and probabilities close to but not equal to 1 from certainty; for example, 0.001 and 0.999 respectively.
  Why it matters: This prevents treating unlikely events as impossible or likely events as guaranteed.

Key terms: impossible event, probability, certitude, improbable events, highly probable events, nonzero probabilities

5.3.2 Exhaustive and mutually exclusive events

This subsection defines exhaustive and mutually exclusive events using quantized weight ranges and establishes that their probabilities sum to one.

• Exhaustive event coverage: Explain that events are exhaustive when their union covers the entire population or sample space, so every possible outcome belongs to at least one event. In the weight example, every Statsville adult's quantized weight falls into S1, S2, or S3.
  Why it matters: Exhaustiveness ensures that a set of events accounts for every possible outcome.
• Mutually exclusive events: Describe mutually exclusive events as events with null intersections, meaning one population member or outcome cannot belong to more than one of them. For example, a weight in S1 cannot simultaneously be in S2 or S3.
  Why it matters: Mutual exclusivity prevents overlap and lets probabilities of alternative events be added directly.
• Sum rule for alternatives: State that for exhaustive, mutually exclusive events E1 through En, the sum of their probabilities is one. Explain that this is because one and only one of the alternatives must occur with certainty.
  Why it matters: The sum rule is a basic constraint for constructing and checking probability distributions.

Key terms: exhaustive events, union, population space, S1, S2, S3, mutually exclusive events, mutual intersections, null intersections, E1

5.3.3 Independent events

This subsection defines independent events as events for which knowing one provides no information about the other.

• Independent events: Explain that two events are independent when neither event influences the other, so learning that one occurred reveals nothing about whether the other occurred. Describe the Statsville example: weighing less than 60 kg and living within 5 km of the city center are independent because knowledge of either characteristic says nothing about the other.
  Why it matters: Independence tells us when information about one event should not change what we infer about another event.

Key terms: independent events, E1, G1

5.4 Joint probabilities and their distributions

This section defines joint probability and shows how the joint distribution of independent event categories is formed by multiplying their individual probabilities.

• Joint probability meaning: Explain that a joint probability p(E1, G1) is the probability that all named events occur together, such as a resident weighing less than 60 kg and living less than 5 km from the city center.
  Why it matters: Joint probabilities let us describe outcomes involving more than one variable at the same time.
• Independent joint distribution: Describe how to construct and interpret the joint probability distribution for independent weight and distance categories: each cell p(Ei, Gj) equals p(Ei)p(Gj), and all cells together sum to 1.
  Why it matters: This product rule provides the baseline joint distribution when variables do not affect one another.
• Multiple-event product rule: State the general product rule that the joint probability of independent events E1, E2, ..., En is the product of all their individual probabilities, using the product symbol ∏ as shorthand.
  Why it matters: The general rule extends the two-event calculation to any number of independent events.

Key terms: joint probability, p(E1, G1), events occurring together, joint probability distribution, product rule, independent events, p(Ei, Gj) = p(Ei)p(Gj), Cartesian product, ∏

5.4.1 Marginal probabilities

This section shows how to recover an individual event’s marginal probability from a joint probability table by summing over all exhaustive, mutually exclusive values of the other event.

• Marginal probability calculation: Explain how to compute the marginal probability of an event from a joint probability distribution by summing the joint probabilities across every possible value of the other event. Describe this as adding the appropriate row or column of a joint-probability table and placing the result in its margin.
  Why it matters: Marginal probabilities let you obtain probabilities for one variable even when the available data is organized as joint probabilities for two variables.
• Marginalization over exhaustive events: Justify why summing p(Ei,G) over all possible Ei values gives p(G): the Ei events are mutually exclusive and exhaustive, so the sum covers every way G can occur and the E alternatives factor out as a certain event with probability 1.
  Why it matters: This reasoning explains why marginalization is valid rather than being merely a table-addition rule.

Key terms: marginal probability, joint probability distribution, joint probabilities, exhaustive events, mutually exclusive events, certain event, factor out

5.4.2 Dependent events and their joint probability distribution

This section illustrates how a joint probability table represents dependent height and weight events, using a toy case of perfect correlation.

• Dependent height-weight events: Describe why height and weight events in the Statsville example are dependent rather than independent. Explain that knowing a resident's height range helps predict their weight range, unlike the earlier independent weight-and-distance example.
  Why it matters: Dependence is what makes a joint distribution convey predictive relationships between variables.
• Perfect-correlation joint table: Explain how Table 5.7 encodes perfect correlation between weight events Ei and height events Fj. State that all table entries together sum to 1, while every off-diagonal probability p(Ei, Fj) is zero when i differs from j, so each weight category implies its matching height category and vice versa.
  Why it matters: Reading the pattern of probabilities in a joint table lets you identify an extreme dependent relationship directly from data.

Key terms: dependent events, independent events, joint probability distribution, height, weight, p(Ei, Fj), perfectly correlated, Ei, Fj, Table 5.7

5.5 Geometrical view: Sample point distributions for dependent and independent variables

This section visualizes joint discrete distributions as sampled point clouds to contrast independent and dependent variables and relate correlation to dimensionality reduction.

• Joint-distribution bucket visualization: Describe how a two-dimensional discrete joint distribution can be visualized as a grid of buckets, with one bucket for each pair of discrete events. Explain that samples are placed randomly within their selected bucket, so high-probability buckets contain more points than low-probability buckets.
  Why it matters: This visualization turns a joint-probability table into a geometric picture of where paired observations occur.
• Independent and dependent point patterns: Distinguish the geometric sample patterns of independent and dependent variables. State that independent-event samples are diffused across the joint space because all combinations have non-negligible probability, whereas dependent-event samples concentrate in particular high-probability regions, such as along a diagonal.
  Why it matters: The shape of a point distribution provides an intuitive way to recognize whether two variables carry related information.
• Correlation and dimensionality reduction: Explain why highly dependent or correlated variables are candidates for dimensionality reduction: their dimensions essentially contain the same information, so knowing one permits deriving the other and one dimension can be dropped with little significant information loss.
  Why it matters: Recognizing redundant correlated dimensions is the key intuition behind reducing data while retaining its important structure.

Key terms: joint distribution, joint event, bucket, sample point distribution, independent events, dependent events, correlated events, joint probability samples, joint space, principal component analysis (PCA)

5.6 Continuous random variables and probability density

This section extends discrete frequency-based probability distributions to continuous random variables through probability density functions whose total integral is one.

• Continuous density construction: Explain how progressively narrower quantization buckets for a continuous quantity lead, in the limit, to a probability density function p(x). Describe that the frequentist probability for each finite bucket is its population count divided by total population.
  Why it matters: Probability density functions let machine learning represent quantities that can take values at arbitrarily fine resolutions.
• Infinitesimal continuous events: Describe the continuous events {x ≤ X < x + δx} as infinitesimally narrow intervals across all x, rather than ordinary discrete outcomes. Explain that these events are exhaustive over the domain and mutually exclusive, and state the section's nuance that p(x) refers to X lying in an infinitesimal interval rather than exactly equaling x.
  Why it matters: Viewing a continuous distribution as exhaustive, nonoverlapping infinitesimal events explains why the same probability rules can be extended from discrete to continuous variables.
• Density normalization integrals: State that the discrete sum rule becomes an integral for a continuous density: the integral of p(x) from negative infinity to infinity equals one. Extend this to a vector random variable, where the multidimensional integral of p(vector x) over its full domain D equals one.
  Why it matters: Normalization ensures that a density accounts for all possible outcomes and is the foundation for using continuous distributions quantitatively.
• Geometric density interpretation: Describe the geometric meaning of density normalization: the integral is the area under p(x) for one variable and the volume under the hypersurface p(vector x) in higher dimensions. Give the XY plane as the domain of a two-dimensional vector example and note that one integral sign may stand for a multidimensional integral.
  Why it matters: The area-and-volume interpretation provides an intuitive way to check what a normalized probability density represents.

Key terms: quantized random variables, buckets, probability density function, p(x), frequentist approach, random variable, E = limδx→0 {x ≤ X < x + δx}, exhaustive, mutually exclusive, continuous counterparts

5.7 Properties of distributions: Expected value, variance, and covariance

This section introduces expected value, variance, and covariance as common distribution parameters needed to fit and understand probabilistic generative models.

• Distribution parameter significance: Explain why mean, variance, and related properties matter when fitting a parameterized probability distribution to training data in generative machine learning.
  Why it matters: These properties are the interpretable parameters that connect training data to a fitted probabilistic model.
• Common distribution properties: State that this section covers expected value, variance, and covariance as properties or parameters shared broadly across probability distributions, including their geometric significance.
  Why it matters: These common descriptors provide a vocabulary for comparing distributions and interpreting their shapes geometrically.
• Later distribution analysis roadmap: Describe the section's roadmap: later subsections connect common properties to individual distributions and show how to obtain their values programmatically with the PyTorch distributions package.
  Why it matters: The roadmap prepares you to relate mathematical distribution parameters to practical PyTorch computations.

Key terms: generative machine learning models, parameterized distribution, distribution family, training data, mean, variance, expected value, covariance, geometric significance, individual distributions

5.7.1 Expected value (aka mean)

This section defines expected value as a probability-weighted average, extends it to functions and linear combinations of random variables, and relates it to centroids and dot products.

• Expected value as weighted average: Explain expected value as the fixed long-run average obtained by repeatedly sampling a distribution, and as the probability-weighted average of a random variable's possible values. Describe how the formula uses a sum for discrete variables and an integral for continuous variables.
  Why it matters: Expected value provides the mean or central representative value of a probability distribution.
• Expected values of functions: Describe the expected value of f(X) as the probability-weighted sum or integral of the function values f(x) over possible values of X. Explain its dot-product interpretation in the discrete case: high expected values arise when high function values align with high probabilities.
  Why it matters: This extension lets probability distributions summarize not only a variable itself but also quantities computed from it.
• Expected-value linearity: State that expected value is a linear operator: the expectation of a weighted linear combination of random variables equals the same weighted combination of their individual expectations.
  Why it matters: Linearity makes averages of combined random quantities easier to compute from their separate means.
• Centroid analogy: Relate expected value to the center of mass: probabilities play the role of masses, and random-variable values play the role of point locations in a weighted centroid.
  Why it matters: The centroid analogy gives a physical intuition for why likely values pull the expected value toward themselves.

Key terms: expected value, mean, probability-weighted average, discrete random variable, continuous random variable, expected value of a function, f(X), dot product, probability vector, function vector

5.7.2 Variance, covariance, and standard deviation

This section defines variance and standard deviation as measures of spread around a mean, then generalizes directional spread to multivariate data through the covariance matrix.

• Variance and standard deviation: Explain variance as the expected squared distance of sample values from their mean, and standard deviation as the square root of variance. Describe how small variance means points are tightly packed around the expected value, whereas large variance means they are more spread out.
  Why it matters: Variance distinguishes meaningful large-scale variation in data from small variations that may be noise.
• Spread and dimensionality reduction: Describe why measuring spread can support a simpler representation of data: tightly packed points may be approximated by a point, and points tightly packed around a line may be approximated by that line. Explain that small variation is often treated as noise while large variation can capture the main information.
  Why it matters: This is the intuition behind retaining important directions while reducing the dimensionality of data.
• Directional covariance matrix: Trace how multivariate spread in a unit direction is found by projecting a random vector onto that direction and taking the variance of the resulting scalar. Describe the covariance matrix as the direction-independent matrix of expected outer products of centered vectors, for which the directional variance is the quadratic form l̂ᵀℂl̂.
  Why it matters: The covariance matrix compactly records a multivariate distribution's spread in every possible direction.
• Variance expectation identity: State the equivalent variance expression as the expected value of X squared minus the square of the expected value, derived by expanding the squared deviation from the mean. Explain that this follows because the mean is constant and can be taken outside an expectation.
  Why it matters: This equivalent form is often a convenient way to compute or manipulate variance.

Key terms: variance, standard deviation, expected value, mean, spread, dimensionality reduction, PCA, packing density, noise, variation

5.8 Sampling from a distribution

This section explains how samples represent a probability distribution and how sample statistics and the law of large numbers let us estimate population properties from finite data.

• Sample point cloud representation: Describe what drawing samples from a probability distribution produces and how a collection of samples represents the distribution. Explain that higher-probability values occur more often, creating denser regions in a sample point cloud.
  Why it matters: Machine learning commonly learns about an underlying population from the data points sampled from it.
• Sample statistics for populations: Explain why sample mean and sample variance are used when the underlying distribution is unknown, and relate them to the population's true mean and variance. State that known distributions can instead provide these properties through closed-form expressions.
  Why it matters: Real datasets are finite samples, so their aggregate statistics provide practical estimates of population behavior.
• Law of large numbers: Explain the law of large numbers as the tendency for the average of many samples to approach a distribution's expected value, while noting that finite samples provide no exact guarantee. Describe why higher variance requires more samples for a reliable approximation.
  Why it matters: It explains both why data-based averages can support predictions and why a small or highly variable dataset can be misleading.

Key terms: sample, sample point cloud, population, univariate, multivariate, sample mean, sample variance, true mean, closed-form expressions, law of large numbers

5.9 Some famous probability distributions

This introduction previews how the following subsections use PyTorch and formulas to work with common deep-learning probability distributions and compare theoretical and sampled properties.

• Distribution evaluation methods: Describe the two ways the examples evaluate a distribution's probability: a PyTorch distributions function call and direct evaluation of its mathematical formula. State that they should agree, while practical code should use the PyTorch distribution function.
  Why it matters: You need to distinguish checking the mathematics from the recommended way to implement distributions in practice.
• Theoretical and sample statistics: Explain that the examples compute theoretical mean and variance with PyTorch and also compute mean and variance from samples. State that sufficiently large samples should have sample mean and variance close to their theoretical counterparts.
  Why it matters: This comparison connects a distribution's stated properties to what is observed when data is sampled from it.
• Log probability computations: State that machine learning often uses the logarithm of probability, and explain that this simplifies computations because popular probability distributions are exponential.
  Why it matters: Log probabilities are a recurring computational form in machine-learning methods built on probability distributions.

Key terms: PyTorch distributions function call, raw evaluation from the formula, theoretical mean, sample mean, theoretical variance, sample variance, logarithm of the probability, exponential distributions

5.9.1 Uniform random distributions

This section defines univariate and multivariate uniform distributions and derives their density, mean, and variance.

• Univariate uniform density: Describe a univariate uniform distribution on the interval [a,b], including its constant density inside the interval and zero density outside it. Explain why the density is 1/(b-a): the total area under the density curve must equal 1.
  Why it matters: Uniform distributions provide a simple baseline model for continuous uncertainty over a bounded range.
• Uniform mean and variance: State that the expected value of a univariate uniform distribution on [a,b] is the midpoint, (a+b)/2, and that its variance is (b-a)^2/12. Explain that widening the interval spreads samples farther apart and increases variance with the square of its width.
  Why it matters: The mean and variance summarize the center and spread of a uniform source and show how its range determines its variability.
• Multivariate uniform density: Describe a multivariate uniform distribution as a random vector with constant density over a multidimensional box-shaped domain and zero density elsewhere. State that its nonzero density is the inverse of the domain volume V, so the total volume under the density function is 1.
  Why it matters: Multivariate uniform distributions extend the idea of equal density from a line segment to bounded regions of feature space.

Key terms: uniform probability distribution, continuous random variable, probability density function, univariate, interval [a,b], expected value, mean, variance, uniform interval, width

5.9.2 Gaussian (normal) distribution

This section defines univariate and multivariate Gaussian distributions, relates their parameters to moments, and explains how covariance determines the geometry of Gaussian density surfaces and sampled point clouds.

• Univariate Gaussian shape: Describe the univariate Gaussian probability density as a bell-shaped curve centered at μ, with σ controlling its width and rate of decay away from the center. State that its density approaches but never reaches zero as values move far from the mean.
  Why it matters: The Gaussian is a foundational model for continuous quantities and for noise in machine learning.
• Gaussian moments and normalization: State that a univariate Gaussian has expected value μ, variance σ², and standard deviation σ, and explain why the symmetric density peaks at its mean. Describe that the Gaussian density integrates to one, making it a valid probability distribution.
  Why it matters: These relationships let you interpret Gaussian parameters directly as location and uncertainty.
• Multivariate Gaussian parameters: Describe a multivariate Gaussian as a distribution over vector-valued random variables, with mean vector μ and covariance matrix Σ. Explain that μ is the most likely vector value and that the density describes probability within an infinitesimally small volume around a vector point.
  Why it matters: Multivariate Gaussians model correlated, multi-feature data rather than just one scalar measurement.
• Covariance geometry and principal axes: Explain that the covariance matrix Σ controls the shape of the base of a multivariate Gaussian bell and the shape of a point cloud sampled from it. Distinguish circular, coordinate-aligned elliptical, and rotated elliptical cases, and state that Σ's eigenvectors give ellipse-axis directions while its eigenvalues determine axis lengths and maximum or minimum spread directions.
  Why it matters: Covariance geometry connects probability distributions to PCA, feature correlations, and the visible structure of multivariate data.

Key terms: Gaussian distribution, normal distribution, probability density function, μ, σ, asymptotically approaching zero, expected value, mean, variance, standard deviation

5.9.3 Binomial distribution

This section defines the binomial distribution for counting successes across repeated binary trials, derives its probability formula, and states its mean and variance.

• Binomial trial model: Describe a binomial setting as n repeated trials with two outcomes, success or failure, where each trial has the same success probability π and the random variable X counts the successes. State that X can take the integer values from 0 through n.
  Why it matters: This model turns a repeated yes-or-no process into a distribution over possible success counts.
• Binomial probability formula: Explain how to compute the probability of exactly k successes in n trials: count the possible success/failure orderings with the binomial coefficient and multiply by the probability of each ordering, π^k(1−π)^(n−k). Relate the formula to summing the probabilities of all arrangements that contain k successes.
  Why it matters: The formula avoids tediously listing every possible ordering when the number of trials is large.
• Binomial normalization and moments: State that the probabilities for all possible success counts k=0 through n sum to 1, as expressed by the binomial expansion. State that a binomial random variable has expected value nπ and variance nπ(1−π).
  Why it matters: These quantities summarize the typical number of successes and how much that count varies across repeated sets of trials.

Key terms: binomial distribution, trial, success, failure, probability of success, π, binomial coefficient, combination, k successes, n trials

5.9.4 Multinomial distribution

This section defines the multinomial distribution for counts of multiple mutually exclusive classes across repeated trials and gives its probability, mean, variance, and covariance.

• Multinomial trial structure: Describe the conditions modeled by a multinomial distribution: each of n trials selects exactly one of m mutually exclusive and exhaustive classes with class probabilities π1 through πm that sum to 1. Explain that Xi counts how often class Ci occurs, and that each individual count Xi has a binomial distribution.
  Why it matters: Multinomial distributions extend binomial counting from two possible outcomes to the many-class outcomes common in machine learning.
• Multinomial count probability: Explain how the multinomial probability function computes the probability of a specified vector of class counts k1 through km in n trials. State that the counts must add to n and that the calculation combines the class-probability powers with a combinatorial factor for the number of orderings of those counts.
  Why it matters: It lets you assign a probability to an observed combination of class frequencies, such as counts of several kinds of photos in a sample.
• Count-vector moments: State that the expected count for class i is nπi and its variance is nπi(1−πi). Describe the covariance matrix of the count vector: diagonal entries are those variances, while off-diagonal covariances are −nπiπj because increasing one class count leaves fewer trials available for other classes.
  Why it matters: The mean and covariance describe both the typical class counts and the tradeoffs among counts imposed by a fixed number of trials.

Key terms: multinomial distribution, classes, mutually exclusive, exhaustive, random trial, random variables, multinomial probability function, k1,…,km, n trials, multinomial coefficient

5.9.5 Bernoulli distribution

This section defines the Bernoulli distribution as a one-trial binary distribution and derives its mean and variance after encoding outcomes as 1 and 0.

• Bernoulli trial distribution: Describe a Bernoulli distribution as the special case of a binomial distribution with one success-or-failure trial. State that a binary random variable X equals 1 for success with probability π and 0 for failure with probability 1 − π.
  Why it matters: Bernoulli distributions provide the basic probabilistic model for a single binary outcome.
• Bernoulli mean and variance: Explain why success and failure must first be assigned numerical values, conventionally 1 and 0, before an expected value or variance can be computed. State that under this encoding the mean is π and the variance is π(1 − π), while an average such as 0.3 is not itself a possible binary trial outcome.
  Why it matters: The 0/1 encoding turns binary outcomes into numerical random variables whose average and spread can be analyzed.

Key terms: Bernoulli distribution, binomial distribution, success, failure, discrete random variable, π, expected value, mean, variance, success = 1

5.9.6 Categorical distribution and one-hot vectors

This section defines the categorical distribution as a one-trial multinomial distribution, represents its outcomes with one-hot vectors, and expresses its probabilities compactly.

• Categorical distribution definition: Explain that a categorical distribution models the outcome of one random trial among m mutually exclusive and exhaustive classes with class probabilities π1 through πm that sum to 1. Relate it to a multinomial distribution with n = 1 and to a Bernoulli distribution generalized from two classes to any number of classes.
  Why it matters: Categorical distributions are the probability model for choosing exactly one class from many possible classes.
• One-hot outcome encoding: Describe a one-hot vector as an m-element numerical representation of a categorical outcome: exactly one entry is 1 for the class that occurred and every other entry is 0. Apply it to identify the selected class, such as encoding a Marie Curie photo as the vector whose class-2 entry is 1.
  Why it matters: One-hot vectors provide a compact numerical target representation for multi-class machine learning problems.
• Categorical probability and expectation: Describe how the categorical probability function uses the one-hot components as exponents so that the probability of an observed one-hot outcome is the π value for its selected class. State that expectation and variance are not inherently meaningful for class labels, although the expected value of the numerical one-hot vector can be defined as an artificial numerical construct.
  Why it matters: This explains why one-hot encoding lets a multi-class probability model select the probability assigned to the observed class.

Key terms: categorical distribution, multinomial distribution, Bernoulli distribution, mutually exclusive, exhaustive, class probabilities, one-hot vector, class, outcome, categorical probability function

5.10 Summary

This summary recaps probability distributions, sampling and statistical moments, then connects covariance, PCA, and the geometry of Gaussian distributions.