An embedding is a numerical vector that represents a token or other discrete item. In a Transformer, tokenization maps text to token IDs, which index a learned embedding matrix and produce dense vectors. section 3.1.1
The vectors enter the model together with positional information. Adding positional embeddings to token embeddings produces the input matrix for the first Transformer block, allowing token content and sequence position to be processed together. section 3.1.1
Figure 3.3 The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

Trace the complete input path from raw text and subword tokens through integer IDs, learned vector lookup, positional addition, and the first Transformer block. section 3.1.1
Embedding vectors serve several practical roles:
- Represent token content as dense numerical input rather than as raw text. section 3.1.1
- Provide the foundation on which Transformer layers learn contextual and task-relevant representations. section 3.1.1
- Act as a regularization target during NEFTune fine-tuning, where controlled noise discourages memorization and encourages more general representations. section 3.5.4
NEFTune adds a uniformly sampled random noise vector after token-to-embedding conversion. Its magnitude scales as α/√(L×d), using noise alpha, sequence length, and embedding dimension; training applies the noise, while inference leaves embeddings unchanged. section 3.5.4
Embedding regularization
A supervised fine-tuning configuration can enable NEFTune with a noise parameter such as neftune_noise_alpha=5. The resulting perturbation affects training representations without adding noise to inference outputs. section 3.5.4







































































