Text generation uses a model to produce new natural-language text. Given a prompt, a model can predict a probability distribution over the vocabulary for the next token. Select a token, add it to the sequence, and repeat: the output grows one token at a time. section 16.2.3
Continue a prompt
Starting with “A piece of advice,” a miniature GPT model produces a continuation beginning “A piece of advice, and the best way to get a feel for yourself…”. The example also repeats phrases, including “get a sense of what you are doing.” section 16.2.3
Listing 16.9 Simple generation function for the mini-GPT model
generate <- function(prompt, max_length = 64) {
tokens <- as.array(tokenizer(prompt))
prompt_length <- length(tokens)
for (i in seq(from = prompt_length + 1, to = max_length)) {
prediction <- mini_gpt(matrix(tokens, nrow = 1))
prediction <- prediction@r[1, -1]
next_token <- op_argmax(prediction, zero_indexed = TRUE)
tokens[i] <- as.array(next_token)
}
tokenizer$detokenize(tokens)
}
The simple generation function tokenizes a prompt, repeatedly asks the model for its next-token prediction, selects the most likely token, and detokenizes the result. section 16.2.3
The simple loop can be slow because directly calling the model runs an uncompiled forward pass at every step. Using Keras `predict()` enables compilation, but growing the input at each step can trigger recompilation on TensorFlow and JAX. Padding the input to a fixed length avoids those changing shapes; the illustrated call drops from minutes to less than a second. It still recomputes predictions for the whole sequence while using only the position needed for the next token. section 16.2.3
Listing 16.10 Compiled generation function for the mini-GPT model
compiled_generate <- function(prompt, max_length = 64) {
tokens <- as.array(tokenizer(prompt))
prompt_length <- length(tokens)
tokens[seq(prompt_length + 1, max_length)] <- 0L #1
dim(tokens) <- c(1, max_length)
storage.mode(tokens) <- "integer"
for (i in seq(prompt_length, max_length - 1)) {
prediction <- mini_gpt |> predict(tokens, verbose = 0)
prediction <- prediction[, i, ]
next_token <- which.max(prediction) - 1L
tokens[, i + 1] <- next_token
}
tokenizer$detokenize(tokens)
}
The compiled version pads tokens to a fixed length, calls `predict()` at each step, and writes the selected token into the next position. section 16.2.3
Generated text can also supply plausible new examples for text augmentation when labeled text is scarce, though this approach has its own challenges. Other applications include chatbots built with pretrained Transformer models and models that describe images in natural language. section 14.5.3 section 16.0
