YaRN

As explained in Transformers in Action

YaRN is a technique that rescales rotary-position frequencies to extend a model's usable context length beyond its training length.

YaRN (Yet Another RoPE eNlargement) rescales or interpolates rotary-position frequencies so models trained on short contexts can process much longer sequences without retraining. section 9.0 section 9.5.2

YaRN addresses several practical long-context problems:

• Extends context windows from earlier 2,000- or 4,000-token limits to as many as 128,000 tokens or more in model families such as Gemma 3. section 9.5.2
• Reduces the perplexity increase and loss of coherence that can occur when a model exceeds its original training length. section 9.5.2
• Improves the model's ability to locate important tokens in very long inputs, where attention can otherwise spread across many irrelevant positions. section 9.5.2

Figure 9.10 Illustration of the haystack problem. Unmitigated attention (grey) spreads probability mass almost uniformly across the sequence, making it difficult to isolate important tokens. Sharpened attention (black) produces a clear peak at the target position (dashed line), enabling retrieval even within a very long context.

Visualizes the needle-in-a-haystack problem: diffuse attention makes retrieval difficult, while sharper attention isolates the relevant position in a long sequence. section 9.5.2

How Transformers in Action covers it

How deep it goes

Explains frequency adjustment, motivation, and effects but stops short of algorithmic or implementation details.

Examples

Uses Gemma 3 and the needle-in-a-haystack problem to illustrate long-context scaling and retrieval.

What it assumes

Builds from rotary positional embeddings and shorter-versus-longer context lengths.

What you can do after

Apply YaRN to extend a short-context model's usable sequence length without retraining.

How it connects

Connects YaRN with RoPE, positional interpolation, iRoPE, perplexity, attention scaling, and long-context retrieval.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• Rotary Position Embeddinguses · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.