Embedding layer

As explained in How Large Language Models Work

An embedding layer is a neural-network layer that converts discrete token identities and positions into vector representations.

An embedding layer converts discrete token identities into vectors: numerical representations that place related concepts near one another in a learned semantic space. “Dog” and “wolf” can therefore receive more similar representations than “red” and “France.” The layer does not discover timeless truth; its relationships can reflect patterns and biases in the training data. section 3.1.1 section 3.2.1

A transformer also needs to know where each token occurs. The layer creates a word embedding for token identity and a positional embedding for sequence location, then adds them together. Without the positional part, rearranged tokens would look identical even when word order changes the meaning. section 3.2.1

A token passes through the embedding stage in a small pipeline:

• The token identity is mapped to a meaning-oriented vector. section 3.1.1
• The token’s position is mapped to a position vector. section 3.2.1
• The two vectors are added, producing one representation that carries identity and location into the transformer layers. section 3.2.1

Meaning and order

A model can place “doctor” and “nurse” in a relationship shaped by the text used for training, but that relationship may reproduce social stereotypes rather than describe reality. Separately, shuffling the tokens in a sentence demonstrates why positional information is necessary: the same words in a different order are not necessarily the same statement. section 3.2.1 section 3.2.1

Figure 3.7 Word embeddings do not capture the fact that input tokens appear in a specific order. This information is captured by a positional embedding. The position embeddings work the same way as word embeddings and are added together. The resulting combined embeddings have the information the model needs to understand the order of tokens.

Shows how word and position embeddings are added so the resulting vector carries both token identity and sequence order. section 3.2.1

How How Large Language Models Work covers it

How deep it goes

Traces identity and position vectors through addition, but defers how the layers are created and trained.

Examples

Uses dog–wolf similarity, doctor–nurse associations, red–France contrast, and shuffled tokens to illustrate representation and order.

What it assumes

Assumes token identities and basic neural-network layers while building the meaning of vectors and positional information from examples.

What you can do after

Evaluate how a transformer represents token meaning and sequence position without treating learned associations as objective truth.

How it connects

Connects token representations and positional embeddings to transformer layers, semantic relationships, and output layers.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn embedding layer maps discrete integer indices to learned dense vector representations.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second Editionnn.Embedding is a PyTorch module that maps integer token indices to trainable dense embedding vectors.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsAn embedding layer is a neural-network layer that maps input items or tokens to learned numerical vector representations.

Compare how 4 books explain embedding layer →

Related concepts

• embeddingproduces · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• transformer architecturepart of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• autoencoderpart of · An autoencoder is a neural network that encodes input data into an internal representation and decodes it to reconstruct the input or an approximation.
• semanticsused by · Semantics is the meaning and behavior conveyed or assigned to expressions, constructs, representations, or their combinations in a language or symbolic system.
• PyTorchused by · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
• convolutional neural networkpart of · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.