A probability distribution describes how likely different outcomes are. In machine learning, the outcomes might be news categories, the next token in a sentence, or possible points in an image model’s latent space. “Training data distribution” means something different: the range of examples a model encountered while learning. section 4.2.3 section 16.6 section 17.1.2 section 19.1.1
A distribution over 46 classes
For a news classifier, a final Dense layer with 46 units and softmax activation produces 46 scores for each article. Score i is the predicted probability of class i, and all 46 scores sum to 1. Training with categorical_crossentropy pushes this predicted distribution toward the distribution represented by the correct label. Accuracy checks only the first choice; top-3 accuracy also gives credit when the correct class is among the three highest-scoring choices. section 4.2.3
Listing 4.14 Model definition
model <- keras_model_sequential() |> layer_dense(64, activation = "relu") |> layer_dense(64, activation = "relu") |> layer_dense(46, activation = "softmax")
The final layer turns each input into a probability distribution over the 46 classes. section 4.2.3
The same idea serves different jobs:
- Text generation: a language model predicts a distribution over next tokens. Greedy search picks the most likely token; random sampling draws from the distribution; top-k sampling limits the candidates first. section 16.6
- Image generation: a variational autoencoder encodes an image as parameters of a latent distribution—a mean and variance—then samples a point for its decoder. section 17.1.2
- Deployment: a model can struggle when new inputs fall outside its training data distribution, as illustrated by a classifier mistaking a leopard-print sofa for a leopard. section 19.1.1
Figure 17.4 A VAE maps an image to two vectors, z_mean and z_log_var, which define a probability distribution over the latent space, used to sample a latent point to decode.
The diagram distinguishes a sampled latent point from the mean and log-variance that define its distribution. section 17.1.2
