Spam classification

As explained in Build a Large Language Model (From Scratch)

The text classification task of deciding whether an email or message is spam or legitimate.

Spam classification decides whether an email or message is spam or legitimate. A pretrained GPT model learns this task through supervised fine-tuning rather than instruction prompting alone. section 1.1 section 6.4

Adapt the model by replacing its vocabulary-sized output layer with a two-class head for spam and not spam. Freeze most parameters, then train the new head together with the final transformer block and final LayerNorm. section 6.5

The classification flow is:

• Load pretrained GPT-2 weights into the GPTModel and verify that ordinary text generation remains coherent. section 6.4
• Tokenize each message and pass the resulting sequence through the transformer. section 6.5
• Use the final token representation, which has access to all preceding tokens under causal attention, to produce two logits. section 6.5

Worked tensor flow

For the four-token input “Do you have time,” GPT-2 small produces an output shaped [1, 4, 2]. Selecting the final row yields two class scores, such as [-3.5983, 3.9902], for the spam decision. section 6.5

Figure 6.9 Adapting a GPT model for spam classification by altering its architecture. Initially, the model’s linear output layer mapped 768 hidden units to a vocabulary of 50,257 tokens. To detect spam, we replace this layer with a new output layer that maps the same 768 hidden units to just two classes, representing “spam” and “not spam.”

Use the architecture diagram to compare GPT’s original vocabulary projection with the two-class spam output layer. section 6.5

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Explains model loading, head replacement, parameter freezing, tensor shapes, and last-token selection, without detailing the full training loop.

Examples

Uses a prize-message prompt, a four-token input, two-class logits, and a GPT-2-small architecture.

What it assumes

Assumes familiarity with tokenization and GPT components while explaining classification adaptation and causal attention behavior.

What you can do after

Builds a GPT-based binary classifier by loading weights, replacing the output head, selecting final-token logits, and configuring trainable layers.

How it connects

Connects pretrained weights, fine-tuning, classification heads, transformer blocks, LayerNorm, and causal attention masks.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Related concepts

• fine-tuningused by · Fine-tuning is the further training of a pretrained machine-learning model to adapt its parameters or behavior to a particular task, domain, or objective.
• classifierkind of · A classifier is a predictive model that assigns an input to one or more discrete classes or categories.
• decision treeaddressed by · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
• labeluses · A label is a value or annotation associated with an item that identifies its target category, outcome, or reference answer for learning or evaluation.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.