liveBook · Manning

Positional embedding

As explained in Deep Learning with R, Third Edition

A positional embedding is a representation that encodes an element's location in a sequence and is combined with its content representation.

A positional embedding encodes an element’s location in a sequence and combines that information with its content representation. Transformers need this because attention compares sequence elements without inherently noticing whether a word comes first, last, or somewhere in the middle. section 15.3.4 section 15.3.5

A positional embedding layer typically works as follows:

  • Represent each token with a learned word vector. section 15.3.5
  • Add each position vector to its corresponding token vector to produce a position-aware representation. section 15.3.5

Transformer translation

A translation Transformer applies the positional embedding layer to both English source tokens and Spanish target tokens before passing them to the encoder and decoder. Training uses Adam, sparse categorical cross-entropy, and weighted accuracy; adding positional information produced 67% next-word accuracy. section 15.3.5

How Deep Learning with R, Third Edition covers it

How deep it goes
Explains learned and cosine-based encodings, then implements learned vectors but stops short of mathematical derivation.
Examples
Uses an R/Keras learned layer and an English-to-Spanish translation Transformer as concrete implementation examples.
What it assumes
Assumes token embeddings and Transformer encoder-decoder components while explaining positional information from the basic sequence-order problem.
What you can do after
Build it from scratch by adding learned position vectors to token embeddings in Transformer encoder and decoder inputs.
How it connects
Connects attention, word embeddings, Transformers, RNNs, and machine translation through sequence-order handling.
Cover of Deep Learning with R, Third Edition
liveBookmanning.com
Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 12 books explain positional embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.