Generation collapse is a severe failure in autoregressive text generation, where output becomes incoherent, repetitive, or structurally degenerate. In attention pruning experiments, removing 54% of attention layers caused immediate collapse; removing 11% produced no measurable speedup. section 8.4.2
Assess pruning with both benchmark scores and sustained generation quality:
- Even when static benchmarks miss the damage, removing three attention layers can introduce subtle factual inconsistencies in long-form generation. section 8.4.2
- Short knowledge distillation can restore coherent generation; the described recovery used 10,000 Cosmopedia records and took ten minutes on an A100. section 8.4.2
Figure 8.10 Benchmark scores for Llama-3.2-3B across four configurations. Degradation between 3 and 6 layers removed is negligible across all benchmarks, but Lambada shows progressive sensitivity to attention pruning at all ratios, and collapses almost completely at 15 layers removed, along with the rest of the tasks.
Compare benchmark degradation across attention-pruning configurations; the 15-layer removal configuration collapses on Lambada and the other reported tasks. section 8.4.2
