liveBook · Manning

Autoregressive generation

As explained in Timeless Algorithms, The Foundational Ideas

Autoregressive generation is the process of producing a sequence by predicting one token at a time, conditioning each prediction on tokens already available.

A multistep answer must build on what has already been said. Autoregressive generation produces such a sequence one token at a time: the model predicts a token, appends it to the available context, and uses that expanded context to predict the next. Contextual representation, output logits, softmax, and token selection are repeated as the response grows. section 12.2.3

In a transformer, causal masking prevents a position from attending to later tokens. The mask is added to attention compatibility logits before softmax: allowed positions receive 0, while blocked positions receive −∞. Because e^(−∞) = 0, masked positions receive zero attention weight. Each position can therefore use only itself and earlier context. section 12.2.3

M=[0-∞-∞-∞00-∞-∞000-∞0000]

Each row identifies the position being represented; each column identifies a position it might attend to. The upper triangle blocks future positions. section 12.2.3

A train overtaking problem

A slower train travels for one hour at 100 km/h before a faster train leaves at 120 km/h. A response can establish the 100 km head start, calculate the 20 km/h relative speed, and divide distance by relative speed to obtain five hours. Each generated part becomes context for subsequent tokens. section 12.2.3

Transformer training can calculate representations for many positions in parallel while causal masking preserves the next-token objective. Generation remains sequential: a later token depends on the tokens selected before it. Hardware can accelerate each prediction but cannot remove that dependency, so longer responses increase latency and inference cost. section 12.3.2 section 12.4.2

Successive predictions can draw on relationships across the available context, but availability does not guarantee correct use. Relevant details may receive insufficient attention or be interpreted incorrectly; fluent continuations can also be wrong when learned associations are incomplete or conflicting. section 12.1.2 section 12.4.2 section 12.2.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through causal masking and a multistep example, using simplified operations rather than reproducing a complete trained transformer.
Examples
Uses an overtaking-trains calculation and a four-token attention mask to illustrate chained predictions and available context.
What you can do after
Evaluate and understand how sequential dependencies affect generation, latency, and reliability.
How it connects
Connects causal masking, self-attention, softmax, next-token prediction, and parallel training.

Also explained in

Compare how 3 books explain autoregressive generation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.