Top-k sampling generates text by choosing the next token from only the k highest-scoring candidates; all other tokens are excluded. It offers a middle ground between always choosing the most likely token and sampling from every token. section 16.2.4
Top-k and temperature sampling shape token selection in different ways:
- Top-k excludes candidates outside the top k; temperature changes how likely tokens are without excluding any. section 16.2.4
- The two can be combined, such as sampling from the top five candidates with a temperature of 0.5. section 16.2.4
top_k <- function(preds, k = 5, temperature = 1) {
preds <- preds / temperature
.[top_preds, top_indices] <- op_top_k(preds, k = k, sorted = FALSE)
choice <- random_sample(top_preds)
op_take(top_indices, choice)
}
The R function scales predictions by temperature, selects the top k values with op_top_k(), then samples among them. section 16.2.4
Changing k
With the same prompt, examples using k = 5 and k = 20 produce different generated passages, illustrating how the candidate cutoff shapes output. section 16.2.4
