liveBook · Manning

Projection

As explained in The AI Pocket Book

A linear transformation that maps vectors into another space, used here as part of transformer attention computations.

A projection maps a vector into a different space by multiplying it by a matrix. In practice, it can squeeze a large embedding into fewer dimensions while keeping information useful for a chosen task. section 1.16

Squeezing an embedding

Imagine flattening a 3D shape into a thin plate. A projection could squash a 12,288-dimensional embedding into 100 dimensions focused on ideas such as “Barks,” “Mammal,” and “Pet,” discarding other dimensions. The useful directions are learned rather than designed by hand. section 1.16

A projection plays several roles in AI systems:

  • Transforms an embedding from one vector length to another, such as 10,000 dimensions to 2,000. section 1.25
  • Produces the output scores used to assign a probability to every possible next token. section 1.21
  • Aligns representations from different modalities, such as turning an image embedding into an embedding comparable to an LLM’s “cat” token. section 1.32
import torch
embedding = torch.nn.Embedding(num_embeddings=100000, embedding_dim=10000)
projection = torch.nn.Linear(10000, 2000)
model = torch.nn.Sequential(embedding, projection)

A short PyTorch example defines an embedding, applies a linear projection, and places both operations in sequence. section 1.25

The matrix determines which direction the space is squeezed toward, but the resulting dimensions are not automatically human-readable. Their parameters are learned during training, so interpreting a projected dimension can be difficult. section 1.16 section 1.25

How The AI Pocket Book covers it

How deep it goes
Defines matrix-based dimensionality changes and gives applications, but stops short of deriving the mathematics or interpreting learned dimensions.
Examples
Uses animal-topic dimensionality reduction, a PyTorch linear layer, token probabilities, and image-to-text embedding alignment.
What it assumes
Assumes familiarity with vectors and embeddings while introducing matrix multiplication as the projection mechanism.
What you can do after
Apply it via PyTorch by placing a linear projection after an embedding operation.
How it connects
Connects projections with embeddings, machine learning, transformer architecture, LLMs, CNNs, and multimodal AI.
Cover of The AI Pocket Book
liveBookmanning.com
Read The AI Pocket Book on liveBook Get the book at manning.com

Also explained in

Compare how 6 books explain projection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of The AI Pocket Book. The section links point to the parts of the book it draws on.