liveBook · Manning

Autoregressive generation

As explained in Rearchitecting LLMs

Autoregressive generation produces a sequence one token at a time, conditioning each new token on previously generated tokens.

Autoregressive generation produces text one token at a time: each new token depends on the tokens already generated. section 4.1.2

During generation, each layer retains the Key and Value attention states of prior tokens in the KV cache. The cache grows linearly with sequence length and can exceed the model’s parameter storage. Removing complete blocks removes their attention layers and reduces cache growth; removing 25% of layers reduces growth per additional token by 25%. section 4.1.2

Multiple-choice and binary benchmarks usually require only one generated token, so they may miss degradation in free-form generation. After removing three layers, a 3B model remained coherent on short prompts but showed factual inconsistencies in sustained long-form generation. Text-generation uses therefore need a recovery process, even a brief one, to restore autoregressive coherence. section 8.5.2

How Rearchitecting LLMs covers it

How deep it goes
Defines token-by-token dependence and explains per-layer key/value storage as the cache grows with sequence length.
Examples
Contrasts one-token benchmark answers with short-prompt coherence and long-form factual inconsistencies after layer removal.
What you can do after
Evaluate whether pruned models retain coherent text generation instead of relying only on static benchmarks.
How it connects
Connects autoregressive generation to KV caches, attention layers, depth pruning, and Attention Drop recovery.

Also explained in

Compare how 3 books explain autoregressive generation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.