liveBook · Manning

Query misalignment

As explained in Retrieval Augmented Generation, The Foundational Ideas

Query misalignment is the failure of a multi-token query to produce a single clear attention focus because its tokens attend to different context regions.

Query misalignment occurs when a multi-token question produces competing attention focal points instead of one clear signal. Syntactic tokens may attend to grammatical anchors, while semantic tokens attend to topical content, blurring which context matters. section 13.3

From scattered query tokens to one focal token

A question such as “where is Daniel?” distributes attention across different context regions. An answer hint prefix—“Daniel is in the ___.”—places the answer at one blank token, giving attention a single target for identifying relevant context. section 13.3

Figure 13.2 The two obstacles to reading attention, and the answer hint prefix that resolves them. Long sequences dilute softmax mass; multi-token queries scatter focal points. Reformulating the query as a fill-in-the-blank prefix forces a single focal token whose attention concentrates on the context that informs the prediction.

The figure contrasts query misalignment with attention dilution and shows how an answer hint prefix concentrates attention on one blank-token prediction. section 13.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines competing syntactic and semantic focal points and stops short of formal attention derivations.
Examples
Uses “where is Daniel?” and the answer hint prefix “Daniel is in the ___.”
What it assumes
Assumes familiarity with model attention, tokens, context, and softmax-based scoring.
What you can do after
Evaluate why multi-token attention patterns blur and understand how a blank-token prefix supplies one focal point.
How it connects
Connects query misalignment to self-attention, attention dilution, context pruning, and HyDE.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.