A text classifier assigns an input sequence to a category. It could label a message as spam, identify a review’s sentiment, or sort a document by topic. To use a neural network, the text first needs a numerical representation the network can process. section 9.7.1 section 4.5
The representation matters. A simple bag-of-words approach averages word embeddings, but does not account for word positions. Models that use positions need a more advanced design. A transformer encoder takes a token sequence and produces a fixed-size representation useful for classification. Unlike a causally masked decoder, it can draw information from all words in the input. section 7.3 section 9.7.1
A practical way to think through the task is:
- Represent each text as numbers suitable for a neural network. section 4.5
- Pair examples with target category labels. In a PyTorch classification dataset, each item can be returned as a sample and its integer label. section 1.2 section 7.1.2
- Choose a model suited to the representation: an averaged-embedding network for a simple baseline, or an encoder when information across the token sequence matters. section 7.3 section 9.7.1
Spam detection
Given a message, the desired result is a category such as spam or not spam. A transformer encoder can turn the token sequence into a fixed-size representation for that classification task. BERT is an example of an encoder model that is pretrained on text and fine-tuned for tasks. section 9.7.1
Figure 9.12 Diagram of the transformer encoder. The only changes from the previous GPT-style decoder are the removal of the masking from the multiheaded attention and the linear layer before output.
The encoder diagram highlights the absence of causal masking, which lets the model use information from across the input sequence. section 9.7.1
