liveBook · Manning

Query

As explained in Timeless Algorithms, The Foundational Ideas

In an attention mechanism, a projected representation of an element that is compared with keys to determine which information to attend to.

When a token such as “it” must draw on surrounding context, a query represents what that token is seeking. In self-attention, the query is compared with keys, which represent how other tokens can be identified as relevant. Their values contain the information they can contribute. section 12.2.1

A query is produced by applying a learned projection matrix, WQ, to a token’s embedding. Separate learned projections produce keys and values, allowing the same token to take different roles in seeking information, signaling relevance, and contributing content. section 12.2.1

A query–key dot product measures vector alignment and supplies a compatibility score. Scores are scaled and passed through softmax to produce attention weights; those weights determine how much each value contributes to the contextual representation. section 12.2.1

Resolving a pronoun

Suppose the query for “it” is [0.2, 0.8], and the keys for “trophy” and “box” are [1, 0] and [0, 1]. Their dot products with the query are 0.2 and 0.8, respectively. After scaling and softmax, the two weights are approximately 0.39 and 0.61, so the box value contributes more to the representation of “it.” This calculation assumes identity projections and uses deliberately simple vectors; the embedding components are not probabilities or necessarily interpretable dimensions in a trained model. section 12.2.1

A single query–key calculation illustrates one contribution to context, not the full behavior of a transformer. Trained models combine many attention heads, feed-forward networks, and stacked layers. section 12.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through projections, dot products, scaling, softmax, and weighted values, while simplifying the behavior of a complete transformer.
Examples
Illustrates queries with pronoun resolution, semantic association, factual recall, and autoregressive generation.
What you can do after
Evaluate how a query affects attention weights and the resulting contextual representation in a simplified example.
How it connects
Connects queries to embeddings, keys, values, self-attention, dot products, and softmax.

Also explained in

Compare how 14 books explain query →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.