Sampling chooses the next token from a language model’s predicted probability distribution. Greedy search always takes the most likely token, while randomized strategies explore alternatives and can avoid repetitive loops. section 16.2.4
Common text-generation strategies include:
- Greedy search selects the highest-probability token at every generation step. section 16.2.4
- Random sampling draws directly from the categorical distribution over the vocabulary, producing more diverse but sometimes incoherent text. section 16.2.4
- Top-k sampling restricts choices to the k most likely tokens before sampling among them. section 16.6
- Temperature sharpens or widens the distribution: low values approach greedy behavior, while high values produce increasingly random output. section 16.2.4
Temperature changes the generated text
With the same prompt, temperature 2 produces seemingly random tokens, 0.8 produces more continuous prose, and 0.2 produces repetitive patterns closer to greedy search. section 16.2.4
compiled_generate <- function(prompt, sample_fn, max_length = 64) {
tokens <- as.array(tokenizer(prompt))
prompt_length <- length(tokens)
tokens[seq(prompt_length + 1, max_length)] <- 0L
dim(tokens) <- c(1, max_length)
storage.mode(tokens) <- "integer"
for (i in seq(prompt_length, max_length - 1)) {
prediction <- predict(mini_gpt, tokens, verbose = 0)
prediction <- prediction[, i, ]
next_token <- sample_fn(prediction) - 1L
tokens[, i + 1] <- as.array(next_token)
}
tokenizer$detokenize(tokens)
}
A generation loop tokenizes a prompt, repeatedly predicts the next-token distribution, applies a sampling function, and appends the selected token. section 16.2.4
