Text generation produces output from a prompt by preparing input tokens, running a pretrained language model, selecting tokens, and decoding them into text. section 2.1 section 4.4.1
A practical generation loop follows these steps:
- Tokenize the prompt into token IDs and pass the sequence to the model. section 3.3
- Read the logits for the final input position, then select the highest-scoring vocabulary entry as the next token. section 4.4.1
- Feed generated tokens back into the loop until reaching the token limit or end-of-sequence marker, while optionally streaming decoded text. section 2.8 section 3.3
For longer outputs, KV caching stores attention keys and values so each iteration processes the new token instead of recomputing the unchanged prefix. section 2.8
Selecting “Berlin”
Given “The capital of Germany is”, tokenization produces IDs; the model scores the vocabulary, and argmax selects token 19846, decoded as “ Berlin”. section 4.4.1
Figure 2.2 An overview of an LLM generating a response (output text) given a user query (input text)
Use the pipeline diagram to trace movement from user input through the model to generated output. section 2.1
