liveBook · Manning

Generation collapse

As explained in Rearchitecting LLMs

Generation collapse is a failure mode in which autoregressive text generation rapidly becomes incoherent, repetitive, or structurally degenerate.

Generation collapse is a severe failure in autoregressive text generation, where output becomes incoherent, repetitive, or structurally degenerate. In attention pruning experiments, removing 54% of attention layers caused immediate collapse; removing 11% produced no measurable speedup. section 8.4.2

Assess pruning with both benchmark scores and sustained generation quality:

  • Even when static benchmarks miss the damage, removing three attention layers can introduce subtle factual inconsistencies in long-form generation. section 8.4.2
  • Short knowledge distillation can restore coherent generation; the described recovery used 10,000 Cosmopedia records and took ten minutes on an A100. section 8.4.2
Figure 8.10 Benchmark scores for Llama-3.2-3B across four configurations. Degradation between 3 and 6 layers removed is negligible across all benchmarks, but Lambada shows progressive sensitivity to attention pruning at all ratios, and collapses almost completely at 15 layers removed, along with the rest of the tasks.

Compare benchmark degradation across attention-pruning configurations; the 15-layer removal configuration collapses on Lambada and the other reported tasks. section 8.4.2

How Rearchitecting LLMs covers it

How deep it goes
Links severe layer removal to immediate collapse and describes recovery through short knowledge distillation.
Examples
Compares Llama-3.2-3B pruning configurations and reports benchmark collapse after removing 15 attention layers.
What you can do after
Diagnose pruning damage by checking benchmark scores and sustained generation quality.
How it connects
Links generation collapse to attention pruning and knowledge distillation.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.