liveBook · Manning

Training

As explained in The AI Pocket Book

The process of fitting a model using data and computation.

Training is the process of filling a model’s parameter blanks so it performs a chosen task. An engineer designs the architecture; an algorithm uses data to adjust the parameters and improve performance. section 1.25

For an LLM, training can shape the embedding space itself. Token vectors begin as unspecified values, and learning organizes them toward the goal of guessing the next word effectively. section 1.19

The basic workflow is straightforward, even when the scale is not:

  • Design a model architecture by arranging building blocks for the application. section 1.25
  • Leave parameters unspecified rather than hand-writing their values. section 1.19 section 1.25
  • Run a data-guided algorithm that searches for parameter settings that improve the task result. section 1.25
  • Account for substantial time, data, electricity, and other resource demands. section 6.9

A small model template

A PyTorch model can define an embedding operation, a projection, and their sequential composition while leaving millions or billions of parameter values for training to determine. section 1.25

import torch
embedding = torch.nn.Embedding(num_embeddings=100000, embedding_dim=10000)
projection = torch.nn.Linear(10000, 2000)
model = torch.nn.Sequential(embedding, projection)

Inspect a compact PyTorch architecture that separates hand-designed operations from values learned during training. section 1.25

Table 6.1 Average electricity consumption across different models compared with typical household consumption

kWh / 1,000 responses

% of daily household kWh (US)

% of daily household kWh (UK)

Text generation

0.047

0.15%

0.5%

Image generation

2.907

9.83%

30%

Compare measured electricity consumption for text and image generation across 1,000 model uses. section 6.9

How The AI Pocket Book covers it

How deep it goes
Explains architecture, parameters, data-guided adjustment, and resource demands without detailing the optimization algorithm’s internals.
Examples
Uses token-to-vector dictionaries, a PyTorch embedding-and-projection model, and measured electricity comparisons for generated outputs.
What it assumes
Builds from the distinction between hand-written software, model architecture, parameters, and data-guided learning.
What you can do after
Evaluate how a model’s architecture and data-guided parameter fitting produce task performance and resource costs.
How it connects
Connects training with machine learning, model architecture, parameters, embeddings, transformer architecture, and next-word prediction.
Cover of The AI Pocket Book
liveBookmanning.com
Read The AI Pocket Book on liveBook Get the book at manning.com

Also explained in

Compare how 24 books explain training →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of The AI Pocket Book. The section links point to the parts of the book it draws on.