liveBook · Manning

Embedding

As explained in Building Specialized AI Systems

An embedding vector is a numerical vector representing the semantic or syntactic properties of an input token or other discrete item.

An embedding is a numerical vector that represents a token or other discrete item. In a Transformer, tokenization maps text to token IDs, which index a learned embedding matrix and produce dense vectors. section 3.1.1

The vectors enter the model together with positional information. Adding positional embeddings to token embeddings produces the input matrix for the first Transformer block, allowing token content and sequence position to be processed together. section 3.1.1

Figure 3.3 The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

Trace the complete input path from raw text and subword tokens through integer IDs, learned vector lookup, positional addition, and the first Transformer block. section 3.1.1

Embedding vectors serve several practical roles:

  • Represent token content as dense numerical input rather than as raw text. section 3.1.1
  • Provide the foundation on which Transformer layers learn contextual and task-relevant representations. section 3.1.1
  • Act as a regularization target during NEFTune fine-tuning, where controlled noise discourages memorization and encourages more general representations. section 3.5.4

NEFTune adds a uniformly sampled random noise vector after token-to-embedding conversion. Its magnitude scales as α/√(L×d), using noise alpha, sequence length, and embedding dimension; training applies the noise, while inference leaves embeddings unchanged. section 3.5.4

Embedding regularization

A supervised fine-tuning configuration can enable NEFTune with a noise parameter such as neftune_noise_alpha=5. The resulting perturbation affects training representations without adding noise to inference outputs. section 3.5.4

How Building Specialized AI Systems covers it

How deep it goes
Explains token lookup, positional combination, and NEFTune noise scaling, but stops short of embedding-matrix training internals.
Examples
Uses subword-to-vector input processing and a NEFTune trainer configured with neftune_noise_alpha=5.
What it assumes
Assumes tokenization and Transformer terminology while teaching vector lookup and positional combination from the input pipeline.
What you can do after
Apply embeddings through a Transformer input pipeline and configure NEFTune to regularize instruction fine-tuning.
How it connects
Connects embeddings to tokenization, positional embeddings, Transformer blocks, self-attention, and fine-tuning.

Also explained in

Compare how 72 books explain embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.