Positional interpolation

As explained in Transformers in Action

Positional interpolation is a technique that rescales or interpolates positional representations so a model can operate on sequences longer than those seen during training.

Positional interpolation (PI) rescales or interpolates RoPE frequencies so a model trained on shorter contexts can process much longer sequences without retraining. section 9.5.2

By stretching position information while remaining grounded in the training range, PI helps preserve coherence and limit the perplexity increase that commonly appears beyond the original context length. section 9.5.2

Extending context windows

PI and related refinements support model families such as Gemma 3 in scaling from earlier 2,000- or 4,000-token limits to 128,000 tokens or more. section 9.5.2

Figure 9.10 Illustration of the haystack problem. Unmitigated attention (grey) spreads probability mass almost uniformly across the sequence, making it difficult to isolate important tokens. Sharpened attention (black) produces a clear peak at the target position (dashed line), enabling retrieval even within a very long context.

The haystack illustration shows why long-context extensions must preserve focused attention rather than merely accept more tokens. section 9.5.2

How Transformers in Action covers it

How deep it goes

Explains frequency interpolation, motivation, and perplexity effects but stops short of implementation formulas or procedures.

Examples

Uses Gemma 3 context scaling and long-context retrieval as concrete illustrations.

What it assumes

Assumes familiarity with RoPE and shorter-context model training while explaining the extension problem from its practical motivation.

What you can do after

Evaluate why positional interpolation supports longer contexts without retraining and why perplexity remains an important check.

How it connects

Connects RoPE with YaRN, iRoPE, perplexity, attention temperature scaling, and the needle in a haystack problem.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• Rotary Position Embeddinguses · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.
• context windowproduces · A context window is the bounded sequence of tokens or other input information a model can access when processing or generating an output.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.