Chapter 16: Text generation

Introduction

The chapter introduces generative AI as a way to produce content from learned patterns while emphasizing human creative direction and practical uses beyond art.

• Statistical Patterns and Generation: Explain how deep learning models generate content by learning statistical structure in images, music, or stories and sampling from what they have learned.
  Why it matters: This is the chapter’s basic explanation of where generated content comes from.
• Human Direction and Artistic Meaning: Distinguish the model’s act of generating content from the human interpretation and direction that can make it meaningful art.
  Why it matters: It clarifies why the section presents generative AI as a creative aid, not an artist in its own right.
• Professional Uses Beyond Art: Describe the kinds of routine content tasks the section identifies as suitable for generative AI and why.
  Why it matters: It shows that the proposed value of generative AI is not limited to making artwork.

Key terms: statistical structure, statistical latent spaces, sampling, algorithmic generation, augmented intelligence, latent space sampling, pattern recognition, rote tasks

16.1 A brief history of sequence generation

This section traces sequence generation from early recurrent-network work to GPT, explaining how general pretraining and increasing scale led to stronger text generation and prompting, while noting persistent limitations.

• Early sequence generation milestones: Describe how sequence generation preceded GPT, from LSTM-based music generation to recurrent-network generation of human-like handwriting.
  Why it matters: GPT belongs to a longer history of models generating sequences, not just text.
• GPT pretraining and reuse: Explain the training setup behind the first GPT and why its results were notable across text tasks.
  Why it matters: It explains why broadly pretrained language models became useful beyond generating text.
• Scaling and prompting progression: Trace how increasing GPT scale coincided with better generated text and a shift from fine-tuning GPT-1 to prompting GPT-2 with a few examples and GPT-3 with a task description.
  Why it matters: It connects the models' increasing scale to the changing ways people could use their generated output.
• Persistent GPT-3 limitations: Describe the limitations the section attributes to GPT-3 despite its useful generative output.
  Why it matters: Strong generated text should not be mistaken for reliability or broad adaptability.

Key terms: LSTM, recurrent networks, sequence data generation, GPT, Generative Pretrained Transformer, unsupervised pretraining, fine-tuned, next token, GPT-1, GPT-2

16.2 Training a mini-GPT

This section prepares a manageable portion of the C4 corpus for mini-GPT pretraining by tokenizing documents and building an efficient pipeline of training and validation batches.

• C4 corpus and training scale: Explain why the example uses only a small portion of C4 and describe the resulting scale of the pretraining data.
  Why it matters: The reduced corpus makes the example feasible without making pretraining a small-data exercise.
• SentencePiece tokenization and reversal: Describe how SentencePiece prepares the text and what its detokenize function adds.
  Why it matters: Pretraining needs numeric tokens, while reversing tokens to text makes generated output readable.
• Fixed windows and document boundaries: Explain how the tokenized text becomes training sequences and how document boundaries are represented.
  Why it matters: This explains what constitutes a training sample even when it crosses from one document to another.
• Shard loading and batch readiness: Describe how interleaving, prefetching, and a validation split prepare the dataset for pretraining.
  Why it matters: These steps keep data flowing to the model while reserving batches for a quick validation check.

Key terms: Colossal Clean Crawled Corpus, C4, tokens, SentencePiece, subword tokenization, integer tokens, detokenize(), tf.data, sequences of length 256, document boundaries

16.2.1 Building the model

This section builds a small decoder-only GPT model and explains the architectural choices behind its text generation setup.

• Decoder-only information flow: Explain how GPT simplifies a sequence-to-sequence Transformer by removing the encoder, and describe the resulting left-to-right information flow.
  Why it matters: Left-to-right attention is the defining constraint of the model built here.
• Single-sequence training tradeoff: Explain how a decoder-only model handles a question and answer, and why treating text as single sequences makes pretraining simpler despite limiting expressivity.
  Why it matters: This tradeoff explains why a decoder-only model can be pretrained on large amounts of unpaired text.
• Stacked decoder construction: Describe the changes made to the earlier Transformer decoder and how those layers form the mini-GPT model.
  Why it matters: These changes turn the earlier decoder into the repeated block used by this mini-GPT.
• Tied input and output embeddings: Explain how the model reuses its token embedding weights to produce output token predictions and why the section proposes this reuse.
  Why it matters: Weight tying avoids maintaining separate large matrices for input token embeddings and output predictions.
• Mixed-precision training setting: State why mixed precision is enabled for this model, at the level explained in the section.
  Why it matters: It explains a training-speed choice in the model setup without requiring the later precision details.

Key terms: decoder-only, encoder, left to right, attention, single sequence, latent space, pretraining, bidirectional, cross-attention, dropout

16.2.2 Pretraining the model

Explains how learning-rate warmup stabilizes mini-GPT pretraining, why its output is treated as logits, and what its training results indicate.

• Learning rate warmup: Explain why the mini-GPT training run gradually increases its learning rate and what that does to early parameter updates.
  Why it matters: It explains a practical choice that helps make this Transformer training run stable.
• Logits and classification loss: Explain what the model's output logits represent, how softmax turns them into probabilities, and why the loss uses from_logits=True.
  Why it matters: It clarifies what the model outputs and why its loss is configured to accept those outputs.
• Training budget and results: Describe the mini-GPT pretraining run and explain what its validation results suggest about further training.
  Why it matters: It keeps the reported accuracy in perspective and explains why more compute could improve the model.

Key terms: exploding gradients, warmup steps, LearningRateSchedule, logit, softmax, SparseCategoricalCrossentropy(from_logits=True), validation loss, validation set, undertrained, training steps

16.2.3 Generative decoding

This section explains how mini-GPT generates text one token at a time and why compilation, fixed input shapes, and key-value caching make generation faster.

• Token-by-token generative decoding: Trace how a prompt becomes a generated sequence: the model predicts next-token probabilities, the most likely token at the last position is appended, and the process repeats.
  Why it matters: This is the basic generation loop whose speed the rest of the section improves.
• Compiled fixed-shape generation: Explain why direct model calls make the generation loop slow and how using predict() with padded, constant-length inputs avoids repeated unoptimized execution or recompilation.
  Why it matters: Keeping the input shape fixed lets generation benefit from compiled computation rather than paying a compilation cost at every step.
• Key-value caching for generation: Explain why recomputing the entire sequence at every generation step is wasteful and how caching past attention keys and values allows a causal Transformer to compute outputs for one new position at a time.
  Why it matters: Caching removes redundant work that becomes especially costly as generated sequences grow long.

Key terms: prompt, probability distribution, next token, predict(), compilation, input shape, padding, recompilation, cached generation, key and value vectors

16.2.4 Sampling strategies

This section explains how sampling choices shape mini-GPT’s generated text, from repetitive greedy output to more varied but sometimes incoherent alternatives.

• Greedy Search and Repetition: Explain why always choosing the most likely next token can make generated text repeat itself, even though the model predicts a distribution over tokens.
  Why it matters: It explains why a reasonable-looking token choice at each step can produce poor text overall.
• Random Sampling and Temperature: Describe how sampling from the token distribution changes generation, and explain how low and high temperatures affect the output.
  Why it matters: Temperature lets you adjust the trade-off between predictable repetition and excessive randomness.
• Top-K Candidate Restriction: Explain how top-K sampling limits token choices, how it differs from lowering temperature, and why the two can be combined.
  Why it matters: It distinguishes two controls that can both favor likely text but do so in different ways.
• Mini-GPT Output Limits: Distinguish more plausible sampled text from useful output, and explain why the section expects much greater scale to improve generation quality.
  Why it matters: Better sampling alone does not turn a lightly trained mini-GPT into a useful modern LLM.

Key terms: greedy search, most likely next token, probability distribution, categorical distribution, temperature, logits, softmax, top-K sampling, K candidates, mini-GPT

16.3 Using a pretrained LLM

This section explains why pretrained LLMs are widely reused instead of trained from scratch, then places the book’s chosen Gemma model in scale relative to the earlier mini-GPT.

• Reasons for pretrained model reuse: Explain why most organizations use pretrained LLMs rather than pretraining their own Transformers, citing both cost and electricity use.
  Why it matters: It explains why using an existing model is a practical alternative to training one from scratch.
• Gemma and mini-GPT scale: Describe the scale of the Gemma variant chosen for the example and compare its pretraining data with the earlier mini-GPT.
  Why it matters: The comparison shows how large even the book’s accessible pretrained-model example is beside its mini-GPT.

Key terms: pretraining, Transformer, pretrained models, Gemma, parameters, tokens, mini-GPT

16.3.1 Text generation with the Gemma model

This section uses KerasHub to generate text with pretrained Gemma and explains why its fluent output can still be unhelpful, unpredictable, or false.

• Gemma loading and generation: Describe how KerasHub’s CausalLM provides a pretrained Gemma model and generates text without a hand-written generation routine.
  Why it matters: This is how the example moves from a pretrained model to generated text without rebuilding its components.
• Next-token output behavior: Explain why Gemma’s output can be coherent yet not particularly useful.
  Why it matters: Fluent text alone does not mean the model is trying to give the answer a user wants.
• Prompting as likely continuation: Explain how a longer, more specific prompt can steer Gemma’s output without implying that it interprets the prompt like a person.
  Why it matters: Thinking in terms of likely continuations explains both what prompting can accomplish and what it cannot guarantee.
• Unpredictability and hallucinations: Describe the limits of prompting, why Gemma can produce fabricated claims, and what kind of additional training the section proposes to help.
  Why it matters: It explains why plausible-sounding generations need scrutiny and why the chapter turns to fine-tuning.

Key terms: KerasHub, CausalLM, tokenizer, generate(), sampling strategies, pretraining objective, guess the next word, mini-GPT, autocomplete, prompt

16.3.2 Instruction fine-tuning

The section explains how instruction-response examples fine-tune a pretrained language model and how preprocessing limits training loss to the response.

• Instruction response training sequences: Describe how an instruction and its desired response become one training sequence, with consistent markers identifying their boundaries.
  Why it matters: The training examples show the model both what a user asks and what kind of answer should follow.
• Fine tuning versus pretraining: Explain why instruction fine-tuning can use the same next-token prediction loss as pretraining while serving a different purpose.
  Why it matters: This distinguishes adapting a pretrained model’s responses from building its language ability in the first place.
• Response only loss weighting: Explain what the CausalLM preprocessor produces and why training loss is computed for response tokens rather than prompt or padding tokens.
  Why it matters: The model should be trained on the answer it is meant to generate, not on fixed prompt text or artificial padding.

Key terms: instruction fine-tuning, input/output pairs, [instruction], [response], guess the next word, pretraining, latent space, CausalLM, preprocessor layer, backbone model

16.3.3 Low-Rank Adaptation (LoRA)

The section explains why full LLM fine-tuning exhausts GPU memory, how LoRA reduces trainable parameters, and what instruction-tuning the adapted Gemma model does and does not achieve.

• Fine-tuning memory bottleneck: Explain why a model that fits in GPU memory for generation may run out of memory during fine-tuning, and how freezing parameters helps.
  Why it matters: It explains why loading and generating with an LLM is possible even when ordinary fine-tuning will not fit on the same GPU.
• Low-rank weight updates: Describe how LoRA adapts a projection while keeping its original kernel frozen, and explain the parameter-count trade-off of a low rank.
  Why it matters: The small trainable update is what makes adapting a large pretrained model practical with limited accelerator memory.
• Attention projection LoRA setup: Describe where the section applies LoRA in the Transformer and how KerasHub enables or customizes that choice.
  Why it matters: It connects the low-rank idea to the specific attention weights adapted in the example.
• Trainable parameter savings: Explain how the LoRA setup changes memory use even though the pretrained model's weights remain present.
  Why it matters: The saving comes from training far fewer parameters, not from removing the pretrained model.
• Instruction-tuning results and limits: Describe the reported changes after instruction-tuning Gemma and why the improved responses do not establish that hallucinations are solved.
  Why it matters: It separates improved instruction-following from reliable factual answers.

Key terms: Adam optimizer, gradients, optimizer state, frozen parameters, LoRA, kernel matrix, low rank, alpha, beta, inner rank

16.4 Going further with LLMs

Introduces upcoming extensions and improvements to basic language modeling without explaining them yet.

16.4.1 Reinforcement Learning with Human Feedback (RLHF)

The section explains how RLHF uses human response rankings to improve an instruction-tuned language model, then illustrates a chatbot trained with preference tuning and a memory-saving choice for loading it.

• Limits of supervised fine-tuning: Explain why handwritten prompt–response examples provide a useful starting point but are insufficient for optimizing chatbot responses at scale.
  Why it matters: This explains why instruction tuning alone is not the end of chatbot training.
• Human-ranked reward model: Describe how human rankings of candidate responses become a reward model and what that model outputs.
  Why it matters: The reward model turns human judgments into feedback that can be used repeatedly during training.
• Reward-guided response training: Trace how generated responses and reward scores can further tune the language model, including the simple example and the possibility of repeating the process.
  Why it matters: It connects the preference scores to the model's subsequent improvement without requiring the details of a reinforcement-learning algorithm.
• Large-model inference precision: Justify the choice of bfloat16 when loading the large Gemma model for inference rather than training it in float32.
  Why it matters: The precision choice makes loading the larger chatbot less memory-intensive.
• Instruction-tuned chatbot behavior: Describe what the Gemma chatbot example shows about prompt formatting, extensive fine-tuning, and the limits of its improved answers.
  Why it matters: The example shows both the practical value of extensive tuning and why one successful refusal is not proof of reliability.

Key terms: supervised fine-tuning, human preference, RLHF, ranked preference dataset, reward model, prompt-response pair, action, reward, reward cutoff, iterative preference tuning

16.4.2 Multimodal LLMs

The section explains how Gemma incorporates images into a text-generating Transformer and places multimodal LLMs within the broader category of foundation models.

• Images as Transformer sequences: Explain why a Transformer can process images when an image encoder converts them into a sequence of vectors compatible with the model’s text representations.
  Why it matters: This is the basic reason the existing Transformer architecture can accept another modality.
• Image soft-token insertion: Trace how Gemma’s image encoder turns an image into soft tokens and how those tokens enter a prompt containing text.
  Why it matters: It explains what it means, concretely, to combine image and text inputs in one sequence.
• Mixed-modality training objective: Describe how the section proposes training with mixed image and text inputs while keeping the language-model prediction objective.
  Why it matters: It clarifies what the model predicts—and does not predict—when images appear in its input.
• Foundation models and reuse: Explain what makes a model a foundation model and why specializing one can be preferable to training a task-specific model from scratch, including the stated drawback.
  Why it matters: The term explains the shared training-and-reuse pattern behind LLMs and other broadly trained models.

Key terms: Transformer, modalities, sequence representation, image encoder, patches, soft tokens, hard tokens, <start_of_image>, token embedding layer, pretraining

16.4.3 Retrieval Augmented Generation (RAG)

This section explains how retrieval-augmented generation adds retrieved information to an LLM’s prompt, why that helps, and how a vector database can find relevant text.

• Reasons for retrieved context: Explain why an LLM used like a search engine benefits from retrieved context: it may invent facts, its knowledge has a cutoff, and relevant context can provide newer or private information while reducing—but not eliminating—hallucinations.
  Why it matters: This explains why supplying information at question time can be more useful than relying on the model’s memory alone.
• RAG question answering flow: Describe how RAG uses a user question to retrieve additional text, then places that text in the prompt so the LLM can answer using it.
  Why it matters: The retrieval step supplies the context that the model can use when composing its answer.
• Vector database retrieval: Trace the vector-database approach described here: embed source documents as vectors, embed the user query, find nearby document vectors, and return their corresponding text.
  Why it matters: This is one way to select which source text gets added to a RAG prompt.

Key terms: cutoff date, private data, hallucinations, factually ground, retrieval-augmented generation, RAG, query, text context, prompt, vector database

16.4.4 “Reasoning” models

This section explains how writing intermediate steps can improve LLM answers and how verifiable answers can be used to train that behavior.

• Chain-of-thought prompting: Explain why prompting an LLM to show its work can improve its answers to math problems, even when it has struggled with similar problems before.
  Why it matters: It explains why asking for intermediate steps can change the quality of an answer without changing the question.
• Iterative chain-of-thought training: Describe the simple training cycle proposed for improving a model's worked solutions to math problems.
  Why it matters: The training cycle shows how checked answers can supply new examples of the step-by-step behavior being sought.
• Verifiable feedback for learning: Explain why answer checking makes the proposed process a reinforcement learning algorithm and how the idea extends to coding.
  Why it matters: Verifiable answers provide feedback even when no human has written out the desired response.
• Longer worked solutions: Describe the trend in how models present their work as they learn to solve harder questions.
  Why it matters: It connects longer generated workings with the section's account of improved problem solving.

Key terms: chain-of-thought prompting, showing your work, intermediate steps, chain-of-thought fine-tuning, supervised examples, string parsing, supervised fine-tuning, reinforcement learning, environment, actions

16.5 Where are LLMs heading next?

The section explains why LLM progress is likely to involve more than larger models, as compute budgets, deployment costs, data scarcity, and learning efficiency shape what comes next.

• Compute Allocation and Scaling: Explain why a fixed pretraining compute budget can be spent on either a larger model or more training data, and why recent LLMs have not simply grown in parameter count.
  Why it matters: It explains why progress in LLMs cannot be judged by parameter counts alone.
• Deployment Size Tradeoff: Explain why a smaller LLM may be preferable to a better-performing larger one when it is deployed.
  Why it matters: Training performance is only part of a model’s usefulness; people also have to be able to afford to run it.
• Pretraining Data Scarcity: Explain the emerging shortage of high-quality pretraining text and why problem-generating environments are attracting interest as another source of training.
  Why it matters: A shortage of suitable text limits the straightforward strategy of improving LLMs by pretraining on ever more web content.
• Human Learning Efficiency Gap: Explain why the section calls for fundamental research alongside scaling LLMs.
  Why it matters: It identifies a limitation that simply continuing to scale current approaches does not settle.
• Natural Language Interface Value: State the capability the section expects LLMs to provide even as their limitations remain.
  Why it matters: It keeps the section’s cautions about LLMs in perspective by naming a capability the author considers consequential.

Key terms: pretraining, parameters, compute, flops, tokens, deploying, model parameter counts, pretraining data, human-written content, reinforcement learning

16.6 Summary