An autoregressive language model generates text one token at a time. After each prediction, it appends the new token to the sequence and uses the expanded sequence as context for the next prediction. Generation stops at an end-of-sequence token or a predefined token limit. section 2.1.2
Growing the context
Start with “The next day.” If the model predicts “is,” the next input becomes “The next day is.” If it then predicts “bright,” the context becomes “The next day is bright.” Each prediction depends on the prompt and the tokens already generated. section 2.1.2
A simple implementation passes the entire growing sequence through the model at every step. It selects the logits for the last position, chooses a token, and concatenates its ID to the input tensor. This makes the feedback loop explicit, but repeatedly processes earlier tokens. section 2.1.3 section 2.3
Listing 2.1 Visualizing autoregressive generation with GPT-2
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
prompt = "The next day is bright"
inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs.input_ids
print(f"Prompt: '{prompt}'", end="")
for _ in range(20):
outputs = model(input_ids) #A
logits = outputs.logits
next_token_logits = logits[:, -1, :] #B
next_token_id = next_token_logits.argmax(dim=-1).unsqueeze(-1)
input_ids = torch.cat([input_ids, next_token_id],
[CA] dim=-1) #C
new_token = tokenizer.decode(next_token_id[0])
print(new_token, end="", flush=True)
print("\n")
Inspect a PyTorch loop that selects the highest-logit token from pretrained GPT-2 and appends it to the input. section 2.1.3
The redundancy is visible inside attention. When a four-token input grows to five tokens, the earlier query, key, and value projections remain unchanged with fixed inference weights. The top-left 4×4 block of the new attention-score matrix repeats the previous calculation, even though only the last position's context vector is needed to predict the next token. section 2.3.2 section 2.3
A key-value (KV) cache avoids recomputing past key and value projections:
- Prefill: process the prompt in one forward pass and populate the cache. section 2.4.2
- Decode: compute query, key, and value vectors for the new token; append its key and value to the cache; then use the new query with the updated cache to calculate attention and predict the next token. section 2.4.2
- Keep caches per layer and attention head. Reuse reduces computation, but cache memory grows with sequence length, layers, and heads. section 2.4.2 section 2.9
For attention, caching reduces the stated cost of each new-token step from quadratic to linear in sequence length, and total generation from cubic to quadratic. Multi-query and grouped-query attention reduce cache memory by sharing key and value projections, with a trade-off in model expressivity. section 2.9







