Score function

As explained in Inside Deep Learning: Math, Algorithms, Models

A function that computes an attention result from queries, keys, and values using their similarities.

A score function assigns a raw importance value to each input in an attention mechanism. Softmax converts the scores into relative weights, which weight the inputs before they are added into the attention output. section 10.1.2

For inputs x₁,…,xₜ, a feature network first produces hᵢ = F(xᵢ). The score function then evaluates each hᵢ. In the simplest form, scores are computed independently; a contextual score also receives information about the whole sequence, such as the average of its extracted features. section 10.1.2 section 10.2

The attention calculation follows this pipeline:

• Transform each input into a learned feature representation. section 10.1.2
• Use the score function to produce one raw score for every item. section 10.1.2
• Apply softmax across the scores to obtain relative importance weights. section 10.1.2
• Multiply each input by its weight and sum the results into the output. section 10.1.2

Context makes a local scoring decision depend on global information: each item can be judged relative to the other items. In multiheaded attention, related scoring uses queries against keys, with values supplying the returned content; dot, general, and additive scores are common choices. section 10.2 section 10.5 section 12.5.1

Listening in a noisy coffee shop

A listener focuses on a friend’s voice because the surrounding conversations provide context. If the friend is absent, different background sounds may receive higher scores, illustrating why contextual scoring can change attention weights. section 10.2

Figure 10.1 Diagram of how attention mechanisms work. The input sequence x_t can be variable in number (i.e., you could have T = 1 items, T = 13, or T = any number you like). A score function assigns a raw importance value to each item t. The softmax function creates relative scores. The output is then the weighted average of the inputs.

Follow the diagram from raw scores through softmax to the weighted-average attention output. section 10.1.2

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Explains raw scoring, softmax weighting, contextual inputs, and score variants, while deferring detailed score-network implementations.

Examples

Uses a noisy coffee-shop analogy, variable-length input sequences, and query-key-value comparisons with dictionary lookup.

What it assumes

Assumes neural-network feature representations and softmax, while introducing attention scoring from the input-transformation stage.

What you can do after

Evaluate/understand how attention turns learned scores into normalized weights and weighted outputs.

How it connects

Connects score functions to attention mechanisms, softmax, context, multiheaded attention, queries, keys, values, and masking.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Related concepts

• attentionhas part · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• language modelimplements · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• contextuses · Context is the surrounding information, state, or circumstances that condition how an entity, event, or task is interpreted or how a system behaves.
• functionkind of · A function is a defined operation that maps inputs to outputs and, when implemented as a callable program unit, may also produce effects.
• queryuses · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.
• scoreproduces · A numerical measure or rating of the performance, quality, or progress of a result, option, or participant.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.