liveBook · Manning

Positional embedding

As explained in Fight Fraud with Machine Learning

Positional encoding is information added to token representations to indicate the positions or order of tokens in a sequence.

Positional embedding, called positional encoding here, adds position information to token embedding vectors. It gives a transformer a sense of sequence, so tokens are not treated as an unordered collection. section 5.2.1

Ordering a short sentence

For “let us fight fraud,” each word can first become a vector. Adding positional encoding then preserves the order: “let” is followed by “us,” and “us” by “fight.” section 5.2.1

The position-enriched vectors enter the transformer encoder, whose repeated blocks combine multi-head attention and dense layers. Sequence information therefore accompanies tokens before attention processes them. section 5.2.1

How Fight Fraud with Machine Learning covers it

How deep it goes
Defines positional encoding and illustrates its role, without detailing a formula or implementation procedure.
Examples
Uses the token sequence “let us fight fraud” to show vectors and their ordered relationships.
What it assumes
Introduces one-hot token vectors before adding positional information to embedded words.
What you can do after
Evaluate/understand how transformers retain token order after words become embedding vectors.
How it connects
Connects tokenization and embeddings with transformers, multi-head attention, encoder-decoder attention, and masked attention.

Also explained in

Compare how 20 books explain positional embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.