liveBook · Manning

Parameter

As explained in The AI Pocket Book

A learned model value optimized during training rather than fixed by hand.

A parameter is a learned model value optimized during training rather than fixed by hand. Engineers design the model’s operations, but leave numerical blanks for the computer to fill. section 1.25

A billion blanks in an embedding dictionary

Suppose a vocabulary has 100,000 tokens and each embedding has 10,000 numbers. The embedding table contains 100,000 × 10,000 = 1 billion parameters. Training discovers values that organize tokens into a useful embedding space for predicting the next token. section 1.19

Parameters appear throughout the model:

  • Embedding tables store the learned numerical vector associated with each token. section 1.19
  • Projection matrices contain learned values that configure how embeddings are transformed and compared during contextualization. section 1.20
  • Training adjusts parameter values with data to improve performance on the desired task. section 1.25
"a"     -> [?, ?, ?, ?, ?, …]
"b"     -> [?, ?, ?, ?, ?, …]
…
"bark"    -> [?, ?, ?, ?, ?, …]
…
"dog"     -> [?, ?, ?, ?, ?, …]
…

View the embedding dictionary before training, with question marks standing for numerical parameters that have not yet been learned. section 1.19

import torch
embedding = torch.nn.Embedding(num_embeddings=100000, embedding_dim=10000)
projection = torch.nn.Linear(10000, 2000)
model = torch.nn.Sequential(embedding, projection)

See a compact PyTorch model whose embedding and projection blocks contain the parameters that training will set. section 1.25

How The AI Pocket Book covers it

How deep it goes
Explains parameters through embeddings, projection matrices, parameter counts, and training, but avoids optimization derivations.
Examples
Uses token embedding tables, projection layers, billion-parameter counts, and a short PyTorch model as concrete examples.
What it assumes
Assumes basic familiarity with tokens, embeddings, model architecture, and training while explaining learned numerical values from first principles.
What you can do after
Evaluate which model values are learned, estimate their counts, and distinguish parameterized operations from architecture choices.
How it connects
Connects parameters to machine learning, training, embeddings, projection matrices, transformers, and model architecture.
Cover of The AI Pocket Book
liveBookmanning.com
Read The AI Pocket Book on liveBook Get the book at manning.com

Also explained in

Compare how 31 books explain parameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of The AI Pocket Book. The section links point to the parts of the book it draws on.