A generative model learns a space of possible outputs; sampling turns that learned space into a particular output. For text, the model predicts a probability distribution over tokens, and a sampling strategy chooses the next token repeatedly. For images, a sampled point in a latent space can be decoded into a new image. section 16.6 section 17.1.1
One prompt, different continuations
Given “A piece of advice,” greedy search always chooses the most likely next token. Random sampling draws from the predicted distribution instead. In the mini-GPT example, randomness produces more varied text and avoids some repetitive loops, but unrestricted draws can make the continuation jump between unrelated ideas. section 16.2.4
Choose how widely to explore the next-token distribution:
- Greedy search selects the highest-probability token. It is simple, but can repeat high-probability patterns. section 16.2.4
- Random sampling draws from all token candidates. Temperature controls the spread: high values can produce incoherent text; low values approach greedy, repetitive output. section 16.2.4
- Top-K sampling limits the draw to the K highest-ranked candidates. Adjust K and temperature together to balance variety against continuity. section 16.2.4
def top_k(preds, k=5, temperature=1.0): preds = preds / temperature top_preds, top_indices = ops.top_k(preds, k=k, sorted=False) choice = keras.random.categorical(top_preds[None, :], num_samples=1)[0] return ops.take_along_axis(top_indices, choice, axis=-1)
The top-K implementation divides predictions by temperature, keeps K candidates, samples among them, then returns the chosen token’s original index. section 16.2.4
The same idea has a different shape in a variational autoencoder (VAE). An encoder produces a latent mean and log variance for an image. A sampling layer draws normal noise and uses those parameters to obtain a latent point; a decoder maps that point back to an image. For MNIST digits, a two-dimensional latent space also makes it possible to decode a grid of chosen points and inspect how the generated digits change across the space. That grid is a deliberate sweep of coordinates, not a set of random draws. section 17.1.3
Listing 17.2 Latent space sampling layer
from keras import ops class Sampler(keras.Layer): def __init__(self, **kwargs): super().__init__(**kwargs) self.seed_generator = keras.random.SeedGenerator() #1 self.built = True def call(self, z_mean, z_log_var): batch_size = ops.shape(z_mean)[0] z_size = ops.shape(z_mean)[1] epsilon = keras.random.normal( (batch_size, z_size), seed=self.seed_generator #2 ) #2 return z_mean + ops.exp(0.5 * z_log_var) * epsilon #3
The Sampler layer shows the VAE operation directly: draw normal noise, scale it using the encoded log variance, and add the encoded mean. section 17.1.3
Sampling gives control over what a learned distribution produces, not a guarantee that the result is sound. A plausible text continuation may still be unhelpful or factually wrong; a generated image may contain incoherent details. Inspect outputs rather than treating probability as understanding. section 16.6 section 17.1.1
