Autoregressive generation produces text one token at a time: each new token depends on the tokens already generated. section 4.1.2
During generation, each layer retains the Key and Value attention states of prior tokens in the KV cache. The cache grows linearly with sequence length and can exceed the model’s parameter storage. Removing complete blocks removes their attention layers and reduces cache growth; removing 25% of layers reduces growth per additional token by 25%. section 4.1.2
Multiple-choice and binary benchmarks usually require only one generated token, so they may miss degradation in free-form generation. After removing three layers, a 3B model remained coherent on short prompts but showed factual inconsistencies in sustained long-form generation. Text-generation uses therefore need a recovery process, even a brief one, to restore autoregressive coherence. section 8.5.2


