Needle in a haystack test

As explained in Transformers in Action

The needle in a haystack problem is the difficulty of locating a small amount of relevant information within a very long context.

The needle in a haystack problem measures how difficult it is to retrieve a small, relevant signal from an extremely long context. As context length grows, attention can spread across irrelevant tokens and dilute the target signal. section 9.5.2

A buried number

Place the number 42,735 somewhere in a 100,000-token document and ask the model to retrieve it. Without mitigation, attention is nearly flat and the target receives little probability; sharpening creates a distinct peak at its position. section 9.5.2

Attention-temperature scaling and RoPE extensions such as YaRN can sharpen attention and improve retrieval of deeply buried information. The test therefore exposes whether long-context mechanisms preserve useful focus rather than merely accepting longer inputs. section 9.5.2

Figure 9.10 Illustration of the haystack problem. Unmitigated attention (grey) spreads probability mass almost uniformly across the sequence, making it difficult to isolate important tokens. Sharpened attention (black) produces a clear peak at the target position (dashed line), enabling retrieval even within a very long context.

Compare nearly uniform unmitigated attention with the sharp peak produced at the target position. section 9.5.2

How Transformers in Action covers it

How deep it goes

Explains attention dilution, gives a numerical retrieval example, and stops short of specifying an executable benchmark procedure.

Examples

Uses a 42,735 token buried in a 100,000-token document and contrasts flat with sharpened attention.

What it assumes

Assumes attention and token-sequence concepts while explaining the retrieval failure from the underlying attention distribution.

What you can do after

Evaluate long-context retrieval by testing whether a model isolates a target signal within a large input.

How it connects

Connects long-context retrieval with attention temperature scaling, RoPE, YaRN, positional interpolation, and iRoPE.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• context retentionmeasures · Context retention is an AI model's ability to preserve, locate, and accurately recall information from earlier or distant parts of its input context.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.