Context is a numerical representation derived from input text. It summarizes information relevant to predicting what comes next, much like a powerful autocomplete system uses what you have already written. section 1.5
Context fits into the GPT sentence-completion process:
- Tokenize the input into words, word parts, or punctuation marks. section 1.5
- Translate tokens into numbers through embeddings. section 1.5
- Combine the embeddings with context to guess the next best word. section 1.5
- Repeat the process autoregressively, feeding each generated word back into the input to build longer text. section 1.5
A tiny vocabulary
A simplified GPT can represent “The,” “capital,” “of,” “Italy,” “is,” “Paris,” “Rome,” and “London” with numerical embeddings. Given a prompt, its context helps select a plausible continuation. section 1.5
Figure 1.8 Given a prompt, you can calculate the context.
Visualize how a prompt becomes a calculated context before continuation prediction. section 1.5
