liveBook · Manning

Query

As explained in Build a DeepSeek Model (From Scratch)

A query vector is an attention representation that specifies what information a token seeks from other tokens.

A query vector represents what a token seeks from other tokens in attention. It is compared with key vectors, which represent what tokens offer; the resulting attention weights determine how their value vectors contribute to a context vector. section 2.2.1 section 2.4.1

For an input embedding matrix X, a learned projection produces the query matrix Q = XWq. Separate learned projections produce K and V. The score matrix QKᵀ compares each query with the keys; scaling and causal softmax turn the scores into attention weights, which are multiplied by V. section 2.2.1 section 2.3.2

Following one new token

For “The next day is,” let X have shape (4, 8) and Wq shape (8, 4). Then Q has shape (4, 4), and QKᵀ has shape (4, 4). When “bright” arrives, its query has shape (1, 4). Multiplying it by the transposed key matrix for all five tokens gives a (1, 5) score row; its attention weights multiply the (5, 4) value matrix to produce the new token’s context vector. section 2.2.1 section 2.3.2 section 2.4.1

Figure 2.19 The attention weights for "bright" are derived from its query vector and the key vectors of all tokens in the sequence.

Trace how one new token’s query is compared with all token keys to produce its attention weights. section 2.4.1

During autoregressive inference, only the newest token’s context vector is needed for the next prediction. Its query is computed fresh, while previously computed keys and values can be reused instead of projected again. In Multi-Head Latent Attention, the new query is still projected directly, but the new token’s key-value information is first projected into a compressed latent vector. section 2.4.1 section 3.4.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Works through projection shapes, attention scores, causal weights, and new-token computation, including a compressed key-value variant.
Examples
Traces “The next day is” through a four-token projection and computes attention for the newly appended token “bright.”
What you can do after
Understand how to trace a query from an embedding projection through attention scores to a context vector.
How it connects
Connects queries to embeddings, keys, values, attention weights, context vectors, multi-head attention, and the key-value cache.

Also explained in

Compare how 14 books explain query →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.