Query

As explained in Deep Learning with Python, Third Edition

An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.

In attention, a query represents what the model is looking for. It is compared with keys to score relevance; those scores weight the values that are combined. In sequence models, the query can come from the target sequence, while keys and values come from the source. section 15.3.1

Figure 15.6 Retrieving images from a database: the query is compared to a set of keys, and the match scores are used to rank values (images).

A photo-search analogy: compare the query with photo tags (keys), then use match scores to rank the photos (values). section 15.3.1

How Deep Learning with Python, Third Edition covers it

How deep it goes

Defines queries through the search analogy and explains their role in parameterized dot-product attention.

Examples

Uses photo search and target-to-source sequence attention as examples.

What you can do after

Build attention from scratch by scoring queries against keys and using the scores to combine values.

How it connects

Connects queries with keys, values, softmax, and dot-product attention.

liveBookmanning.com

Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Transformers in ActionIn attention, a query is a vector representation that specifies the information for which relevance is being evaluated.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A query is an attention vector representing the information whose relevance to sequence elements is being evaluated.
• 

  liveBookmanning.com
  Sutskever's ListAn attention projection representing the information a sequence position seeks from other positions.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA learned projection of an input position used to score how relevant other positions are in attention.
• 

  liveBookmanning.com
  How Large Language Models WorkAn attention query is a vector representing the information sought when an attention operation compares sequence elements.

Compare how 7 books explain query →

Related concepts

• attentionused by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
• scaled-dot product attentionused by · Scaled dot-product attention computes query-key dot products, divides them by the square root of the key dimension, applies softmax, and uses the resulting weights to combine value vectors.
• Rotary Position Embeddingused by · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.
• keyuses · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.