Attention score

As explained in The AI Pocket Book

A numerical value used by an attention mechanism to weight the relevance of one element relative to others.

An attention score is a number that tells an attention mechanism how relevant each token is to another token. During contextualization, the scores divide attention across the context so the model can resolve meaning. section 1.20

Disambiguating “bark”

In “dog’s bark,” the model gives the most attention to “dog,” then “bark,” and finally “’s.” Information from “dog” therefore influences the embedding for “bark,” making its meaning more specific. section 1.20

The scores come from mathematical operations such as projections that compare embedding vectors. Their detailed calculation is deferred, while learned projection matrices supply many of the numerical settings. Afterward, the scores guide updates to the token embeddings. section 1.20 section 1.24

Figure 1.10 The attention mechanism calculates the relative relevance of all tokens in the context window to contextualize or disambiguate the last token.

A figure showing how the mechanism compares tokens and assigns their relative relevance while contextualizing the final token. section 1.20

How The AI Pocket Book covers it

How deep it goes

Explains token weighting, projections, learned matrices, and embedding updates but defers the detailed mathematical operations.

Examples

Uses “dog’s bark” to show how relative token relevance resolves meaning.

What it assumes

Assumes familiarity with tokens and embedding vectors while introducing attention scores through a concrete language example.

What you can do after

Evaluate how token relevance guides contextualized embeddings in a language model.

How it connects

Connects attention scores to attention mechanisms, embeddings, context windows, projection matrices, and machine learning.

liveBookmanning.com

Read The AI Pocket Book on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Graph Neural Networks in ActionAn unnormalized learned score quantifying the importance of one connected node's features to another node.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn attention score is a numerical value measuring the relevance of one vector or sequence position to another in an attention calculation.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An attention score is a numerical measure of the relevance between a query and a key.

Compare how 4 books explain attention score →

Related concepts

• inner productproduced by · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
• attention weightproduces · An attention weight is a computed coefficient, often normalized, that determines how strongly an input element or value contributes to an attention-based representation.
• queryproduced by · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.
• attentionproduced by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• keyproduced by · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
• softmaxuses · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.

This explainer was generated by Manning's knowledge graph from the text of The AI Pocket Book. The section links point to the parts of the book it draws on.