liveBook · Manning

Training

As explained in Build a Large Language Model (From Scratch)

The process of optimizing model parameters using data so the model learns the desired behavior.

Training optimizes model parameters using data so a GPT model can generate coherent text instead of incoherent output. It operates on GPT architectures built from repeated Transformer blocks, masked multi-head attention, and GELU-based feed-forward networks, after which output tensors are decoded one token at a time. section 4.8

Model scale

GPT models with 124, 345, 762, and 1,542 million parameters can use the same GPTModel Python class, showing that training applies across model sizes. section 4.8

How Build a Large Language Model (From Scratch) covers it

How deep it goes
Connects training to parameter optimization and generation quality but does not detail objectives, data flow, or implementation steps.
Examples
Uses GPT models ranging from 124 million to 1,542 million parameters to illustrate shared implementation across scales.
What you can do after
Evaluate whether training is necessary for coherent text generation from a GPT-like model.
How it connects
Links training to GPT models, Transformer blocks, masked multi-head attention, GELU, token decoding, and text generation.
Cover of Build a Large Language Model (From Scratch)
liveBookmanning.com
Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

Compare how 24 books explain training →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.