Additive score

Q: How deeply does Inside Deep Learning: Math, Algorithms, Models cover additive score?

Explains learned layers, activation, tensor reshaping, and batched computation without comparing empirical performance.

As explained in Inside Deep Learning: Math, Algorithms, Models

An attention scoring method that combines representations through an additive learned function.

Additive scoring lets a small neural network assign each representation a context-dependent attention score. It concatenates an item state hₜ with a context vector, transforms the result with a learned layer and tanh, then produces one scalar through another learned layer. section 10.2.3 section 10.2

A batched implementation follows these steps:

Repeat the context from (B, H) to (B, T, H), matching the sequence of states. section 10.2.3
Concatenate states and context along the feature dimension, producing (B, T, 2H). section 10.2.3
Apply a linear layer from 2H to H, tanh, and a final linear layer from H to 1, returning (B, T, 1). section 10.2.3

PyTorch shape pattern

Use nn.Linear(2*H, H) for the hidden layer and nn.Linear(H, 1) for the output layer; concatenate with torch.cat(..., dim=2) so all T scores are computed at once.

How Inside Deep Learning: Math, Algorithms, Models covers it

Cover of Inside Deep Learning: Math, Algorithms, Models

liveBook manning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Related concepts

linear layeruses · A linear layer is a neural-network layer that maps input features to outputs by a learned affine transformation, typically connecting each output to every input feature.
concatenationuses · Concatenation is the operation of joining ordered sequences end to end to form a single sequence.
tanhuses · The hyperbolic tangent, or tanh, is a smooth nonlinear function that maps real-valued inputs to values between −1 and 1.
score functionkind of · A function used in attention to compute compatibility scores between elements, with variants such as dot, general, and additive.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.