liveBook · Manning

Repetition loop

As explained in Rearchitecting LLMs

A repetition loop is a generation failure mode in which a language model repeatedly produces the same text pattern instead of progressing normally.

A repetition loop is a generation failure in which autoregressive output repeats instead of progressing. Aggressive attention-layer pruning can trigger it immediately. section 8.5.1 section 8.6

Pruning-induced collapse

Removing 15 of 28 attention layers from Llama-3.2-3B yielded about a 6% throughput gain but caused a repetition loop from the first generated token. section 8.5.1

Recovery experiments use knowledge distillation with OptiPFair, varying settings such as alpha/beta balance, calibration sample count, or training duration. Check generated text directly with the Paris prompt; benchmark improvement alone may not establish that generation has recovered. section 8.6 section 8.6

How Rearchitecting LLMs covers it

How deep it goes
Reports immediate collapse after removing 15 of 28 attention layers and proposes varying distillation settings, but gives no recovery results.
Examples
Uses Llama-3.2-3B pruned by 15 of 28 attention layers and the Paris prompt to assess recovery.
What you can do after
Diagnose/debug with it: evaluate recovered generation using the Paris prompt and direct text-quality checks alongside benchmarks.
How it connects
Links repetition loops to attention-layer pruning, autoregressive generation, and knowledge distillation.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.