Chapter 15: Sampling continuous distributions

Introduction

This chapter introduction positions the next two chapters as extending sampling and Bayesian reasoning from categorical distributions to one-dimensional continuous distributions, with this chapter focusing on sampling techniques.

• Continuous-distribution chapter roadmap: Describe how the chapter shifts from categorical distributions to one-dimensional continuous distributions of real numbers, and state that it covers sampling rather than continuous Bayesian inference.
  Why it matters: This roadmap separates the problem of generating continuous samples from the later problem of updating beliefs with continuous evidence.
• Continuous sampling techniques: State that this chapter presents special-purpose sampling techniques for particular continuous distributions, including uniform and normal distributions, the inverse transform algorithm, and rejection sampling.
  Why it matters: These techniques are the chapter's promised tools for producing random real-number samples with specified continuous distributions.
• Division of final chapters: Explain that chapter 15 addresses sampling one-dimensional continuous distributions, whereas chapter 16 applies Bayes' theorem to continuous distributions.
  Why it matters: Keeping the two tasks separate makes clear why the book treats continuous sampling before continuous Bayesian posterior computation.

Key terms: categorical distributions, one-dimensional continuous distributions, real numbers, Bayesian reasoning, uniform distribution, normal distribution, inverse transform algorithm, rejection-sampling algorithm, continuous distributions, sampling

15.1 What is a continuous probability distribution?

This section defines continuous probability distributions and PDFs, explains how probabilities come from relative areas under a PDF, and previews methods for sampling them.

• Continuous distribution support: Explain that a continuous probability distribution has a continuous range of real-number outcomes, so a sample is a real number rather than one item from a discrete set. Describe an example such as a manufactured part’s randomly varying diameter or a modeled probability value.
  Why it matters: Continuous models let programs simulate random processes whose outcomes can take any value in a real-valued range.
• PDFs and relative areas: Describe a probability density function (PDF) as a non-negative real-valued function over real inputs with finite total area. Explain that the probability of a sample falling in a range is that range’s area under the PDF divided by the total area, rather than the height of the curve at one point.
  Why it matters: The area interpretation is the basis for reasoning about continuous probabilities and for the sampling methods that follow.
• Normalization and local density: Explain that a normalized PDF has total area 1.0, so interval probabilities need no further division by total area. Describe the local approximation that the probability of landing between s and s+e for very small positive e is approximately P(s)*e divided by the total area.
  Why it matters: Normalization and local density let you use PDF values correctly even when practical PDFs are not normalized.

Key terms: continuous probability distribution, support, sample, probability density function, PDF, area under the curve, total area, beta distribution, normalized, non-normalized PDF

15.2 Sampling the continuous uniform distribution

This section defines and samples the continuous uniform distribution, introduces continuous-distribution weighting through PDFs, and visualizes samples with vertical histograms.

• Continuous uniform distribution: Describe the continuous uniform distribution as producing a double between a minimum and maximum, with every value in that range equally likely. State that the 0.0-to-1.0 version is the standard continuous uniform distribution.
  Why it matters: This is the basic continuous source of randomness used to construct and sample other continuous distributions.
• Continuous distribution PDF interface: Explain why continuous distributions need interfaces different from enumerable discrete distributions: their support may contain any real number and cannot be listed, and their PDFs are real-valued weight functions. Describe the uniform distribution's PDF as 1.0 between its minimum and maximum and 0.0 elsewhere, noting that a PDF can be non-normalized.
  Why it matters: The PDF-based representation provides the abstraction needed to implement and reason about continuous distributions.
• Vertical sample histograms: Describe how the histogram algorithm visualizes a sequence of double samples by placing samples between low and high edges into one of 40 columns, counting each column, and displaying each as a vertical stack of up to 20 asterisks. Explain that vertical bars make a continuous histogram resemble the PDF, while random sampling causes small deviations from a perfectly flat uniform result.
  Why it matters: A histogram gives a practical visual check that generated continuous samples have the expected shape.

Key terms: continuous uniform distribution, standard distribution, minimum, maximum, support, double-weighted distribution, continuous distribution, PDF, probability density function, non-normalized PDF

15.3 Sampling the normal distribution

This section characterizes normal distributions and shows how Box-Muller sampling of a standard normal can be rescaled to sample any normal distribution.

• Normal distribution characteristics: Describe a normal, or Gaussian, distribution as a symmetric bell-shaped continuous distribution centered at its mean, with its spread determined by standard deviation.
  Why it matters: These two parameters tell you both where normal samples cluster and how widely they vary.
• Normal distribution tails: Explain that a normal distribution's PDF extends infinitely in both directions, while its tails become extremely thin, making samples more than four standard deviations from the mean vanishingly rare.
  Why it matters: This explains why a histogram can focus on the central range even though the mathematical distribution has no finite endpoints.
• Box-Muller normal sampling: Trace how the Box-Muller transform samples a standard normal distribution by using standard-uniform randomness to generate a random radius and angle in polar coordinates, whose Cartesian x coordinate is normally distributed.
  Why it matters: Box-Muller supplies an efficient way to turn uniform random samples into samples from the standard normal distribution.
• Arbitrary normal rescaling: Explain how to sample an arbitrary normal distribution by taking a standard normal sample, multiplying by the desired standard deviation, and adding the desired mean.
  Why it matters: This lets one standard-normal sampler produce samples for normal distributions with any required center and spread.

Key terms: normal distribution, Gaussian distribution, mean, standard deviation, PDF, tails, standard normal distribution, Box-Muller transform, standard uniform distribution, polar coordinates

15.4 The inverse transform method

This section explains how inverse transform sampling turns a standard uniform sample into a sample from any continuous distribution whose quantile function is known.

• Inverse transform sampling procedure: Describe how to sample a continuous distribution with the inverse transform method: draw a probability from the standard continuous uniform distribution and pass it to the desired distribution's quantile function. Explain that this transforms a uniform sample into a sample from the target distribution.
  Why it matters: This gives a concise, efficient way to sample distributions when their quantile functions are available.
• PDF, CDF, and quantile roles: Explain the roles and relationships of a normalized PDF, CDF, and quantile function. State that integrating a normalized PDF produces a nondecreasing CDF, the CDF maps a number to the probability of a sample at or below it, and the quantile function reverses that mapping from probability to number.
  Why it matters: The inverse relationship between the CDF and quantile function is what makes inverse transform sampling work.
• Cauchy distribution example: Describe the Cauchy distribution as the section's example of inverse transform sampling, including that it is bell-shaped but has much fatter tails than a normal distribution. Explain that having its known PDF, CDF, and quantile function makes it straightforward to implement and sample.
  Why it matters: The Cauchy example shows why known quantile functions let useful continuous distributions be represented in only a few lines of code.

Key terms: inverse transform method, standard continuous uniform distribution, quantile function, normalized PDF, cumulative distribution function, CDF, inverse, Cauchy distribution, fat tails, normal distribution

15.5 Rejection sampling

This section adapts rejection sampling to continuous target distributions by using a scaled helper distribution to propose and accept or reject samples.

• Continuous rejection-sampling requirements: Describe the three ingredients needed to sample a continuous target distribution with rejection sampling: its non-normalized PDF, a sampleable helper distribution with equal or larger support, and a scale factor that makes the scaled helper PDF at least as large as the target PDF everywhere.
  Why it matters: These conditions make it possible to generate valid samples without having a direct sampler for the target distribution.
• Dart-based acceptance procedure: Trace how the continuous rejection-sampling algorithm proposes and tests a sample: draw x from the helper distribution, draw y uniformly from zero to the scaled helper function at x, accept x when the point lies below the target PDF, and otherwise repeat.
  Why it matters: The accept-or-reject test converts samples from an easy helper distribution into samples whose histogram follows the target PDF.
• Efficiency tradeoff of rejection: Explain that rejection sampling can perform poorly when many proposed darts are rejected, just as in the earlier categorical version, even though it can sample a target distribution without a specialized direct technique.
  Why it matters: A method that is correct may still be a poor practical choice if its helper distribution causes too many retries.

Key terms: rejection sampling, target distribution, non-normalized PDF, helper distribution, support, scale factor, dart, x coordinate, y coordinate, scaled helper function

15.5.1 Clamping with rejection sampling

This subsection shows how rejection sampling clamps a distribution by discarding samples outside a chosen range, eliminating undesirable tails.

• Clamped distribution sampling: Describe how to clamp a distribution with rejection sampling: draw from the original, unclamped distribution and reject every sample outside the desired range.
  Why it matters: Clamping lets a program preserve a distribution’s behavior in an allowed range while excluding values it cannot or should not use.
• Tail exclusion rationale: Explain why clamping is useful when a distribution has low-probability tails that represent impossible or nonsensical values, such as negative or greater-than-100-percent failure probabilities.
  Why it matters: A mathematically convenient distribution can still produce values that do not make sense for the quantity being modeled.
• Rejection sampler configuration: State that, for clamping, the helper distribution is the original distribution and the rejection-sampling scale factor is 1.0, unlike the earlier two-humped-distribution example.
  Why it matters: This configuration shows that clamping is a particularly direct use of rejection sampling rather than a reshaping scheme requiring a differently scaled helper distribution.

Key terms: clamping, support, rejection sampling, unclamped distribution, clamped range, normal distribution, long, thin tails, impossible regions, low-probability regions, helper distribution

15.5.2 Problems with rejection sampling

This subsection identifies rejection sampling's unbounded runtime and manual helper-fitting limitations, motivating automatic sampling methods for continuous posteriors.

• Unbounded rejection loops: Explain why rejection sampling has no fixed upper bound on the number of attempts needed to produce one sample. Relate a large rejection area to frequent rejections, while noting that even a reasonable acceptance rate cannot guarantee how many loop iterations a particular sample requires.
  Why it matters: An algorithm that can keep rejecting indefinitely is difficult to predict and may be impractical for automated sampling.
• Helper distribution fitting: Describe why choosing a helper distribution and scale factor for rejection sampling is not generally automatic. Explain that a good fit may require expert manual tuning of the helper's family, mean, standard deviation, or scale factor to reduce rejection area, and that target-tail shape can require a different helper such as a Cauchy distribution.
  Why it matters: Manual proposal tuning prevents rejection sampling from being a general automatic method for arbitrary continuous distributions.
• Continuous posterior sampling need: State the unresolved goal of updating beliefs about a continuous phenomenon from observed evidence using Bayes' theorem, thereby sampling a continuous posterior distribution. Explain that the chapter points ahead to chapter 16 for a method intended to sample any continuous distribution, including posteriors.
  Why it matters: Continuous Bayesian models need a practical way to turn updated beliefs into samples.

Key terms: rejection sampling, rejections, acceptance area, rejection area, helper distribution, scale factor, non-normalized PDF, standard normal, Cauchy distribution, fatter tails

15.6 Summary

This summary recaps how PDFs, CDFs, quantile functions, and rejection sampling support sampling from continuous probability distributions.