Average embeddings

As explained in Inside Deep Learning: Math, Algorithms, Models

A sequence-processing technique that replaces a variable-length sequence of embeddings with their mean over the time dimension.

Average embeddings replace a variable-length sequence with one fixed-size representation. For input X of shape (B, T, D), compute the mean across time: X̄ = mean(X, dim=1), producing (B, D). This removes the time dimension so ordinary fully connected layers can process the result. section 12.2

A direct PyTorch implementation follows these steps:

• Map token indices from (B, T) to embeddings with shape (B, T, D). section 12.2
• Apply shared linear layers independently at every time step. section 12.2
• Use nn.AdaptiveAvgPool2d((1, D)) to transform (B, T, D) into (B, 1, D), then flatten to (B, D). section 12.2
• Finish with hidden layers and a classifier. section 12.2

Tradeoff

Averaging is simple and fast, and it works when T changes between batches. It discards order and temporal structure, so faster training can produce worse predictions. section 12.2

Figure 12.1 Example of averaging embeddings over time. The T inputs over time are processed by the same neural network via weight sharing. Because there are no connections across time, the model is unaware that there is an order to them. The time dimension is resolved by simply averaging all of therepresentations so a regular fully connected network can run afterward to produce the output.

Visualizes shared processing across time and the final averaging step before a fully connected network. section 12.2

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Explains tensor shapes and an adaptive-pooling implementation, while stopping short of preserving order or temporal structure.

Examples

Uses a PyTorch embedding classifier, an adaptive average-pooling layer, architecture diagrams, and accuracy-versus-time plots.

What it assumes

Assumes embeddings and tensor dimensions, while demonstrating adaptive pooling and shared linear-layer behavior through implementation.

What you can do after

Build a variable-length sequence classifier using shared layers, adaptive average pooling, flattening, and a final linear prediction layer.

How it connects

Connects embeddings with adaptive pooling, weight sharing, residual connections, recurrent neural networks, and fully connected networks.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Related concepts

• GRUcontrasts with · A gated recurrent unit (GRU) is a recurrent neural-network architecture that uses learned gates to regulate information flow and update a hidden state across sequence steps.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.