Sampling

As explained in Build a Large Language Model (From Scratch)

A method for choosing next tokens stochastically from their probability distribution instead of always taking the highest-probability choice.

Sampling chooses the next token stochastically from a probability distribution rather than always selecting the highest-probability token. It adds variability and creative alternatives to GPT text generation. section 5.3.2 section 4.7

A practical token-generation flow is:

• Convert model logits into probabilities with softmax, optionally applying temperature scaling to make the distribution more or less uniform. section 5.3.2
• Restrict choices with top-k sampling by replacing logits outside the k highest values with negative infinity. section 5.3.2
• Sample one token from the remaining distribution, append it to the context, and repeat the prediction cycle. section 4.7

Top-k sampling with k = 3

For a nine-token vocabulary, the three highest logits are 6.75, 6.28, and 4.51. Masking the other logits produces probabilities of 0.5775, 0.3610, and 0.0615 for the surviving tokens; sampling then selects among these three choices. section 5.3.2

Figure 5.15 Using top-k sampling with k = 3, we focus on the three tokens associated with the highest logits and mask out all other tokens with negative infinity (–inf) before applying the softmax function. This results in a probability distribution with a probability value 0 assigned to all non-top-k tokens. (The numbers in this figure are truncated to two digits after the decimal point to reduce visual clutter. The values in the “Softmax” row should add up to 1.0.)

Shows how top-k masking removes nonselected tokens before softmax, leaving a normalized distribution for stochastic selection. section 5.3.2

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Explains temperature scaling, top-k masking, softmax, and multinomial selection, but does not detail alternative sampling methods.

Examples

Demonstrates a nine-token top-k example and iteratively generates text from the “Hello, I am” context.

What it assumes

Assumes token IDs, GPT logits, and softmax while demonstrating the generation loop from encoded context onward.

What you can do after

Apply it via PyTorch to mask logits, compute probabilities, and select varied next tokens during generation.

How it connects

Connects probabilistic sampling with temperature scaling, top-k sampling, softmax, logits, and greedy decoding.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Related concepts

• determinismcontrasts with · Determinism is the property of a process or system in which identical inputs and relevant states or conditions determine the same outcome or behavior.
• greedy algorithmcontrasts with · A greedy algorithm builds a solution incrementally by choosing the locally best available option at each step, without generally accounting for all future consequences or revising prior choices.
• multinomialuses · Multinomial is a sampling operation that draws one or more categorical outcomes according to specified probabilities or nonnegative weights.
• temperatureused by · A temperature parameter adjusts the concentration of a probability distribution, affecting sampling randomness or the likelihood of accepting less favorable transitions.
• training datasetused by · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.