Chapter 14: Text classification

Introduction

This chapter introduction previews text preprocessing and simple text classification as foundations for later chapters.

14.1 A brief history of natural language processing

The section traces NLP from handcrafted language rules to statistical models and Transformers, while distinguishing successful text pattern recognition from human-like understanding.

• Natural and machine languages: Distinguish natural languages from designed machine languages and explain why natural language is difficult to capture with fixed rules.
  Why it matters: This contrast explains why writing a complete grammar was not a straightforward route to NLP.
• Limits of handcrafted rules: Explain why early rule-based translation successes did not generalize and why NLP researchers began turning to machine learning.
  Why it matters: It explains the shift from manually specifying language rules to learning from text data.
• Statistical nature of NLP: Describe modern NLP as pattern recognition on text and distinguish its practical success from human-like language understanding.
  Why it matters: It gives the right interpretation of what text models accomplish—and what their performance does not establish.
• LSTMs and Transformer scaling: Trace why NLP moved beyond LSTMs toward Transformers and describe the scaling advantage stated in the section.
  Why it matters: It accounts for the architectural shift behind the recent growth of NLP without requiring Transformer mechanics not taught here.

Key terms: natural language, machine language, formal rules, handcrafted rules, context, machine learning, corpus of data, natural language processing, NLP, pattern recognition

14.2 Preparing text data

This section explains how to turn raw text into numeric token sequences through standardization, splitting, and vocabulary-based indexing.

• Text preprocessing pipeline: Explain why text must be preprocessed for deep learning and name the three stages that turn it into numeric input.
  Why it matters: This pipeline connects readable text to the numeric inputs a model can use.
• Character and word tokens: Distinguish character-level from word-level splitting and describe how regular expressions can identify tokens.
  Why it matters: The choice of token units determines what will receive numeric indices.
• Vocabulary indexing and unknowns: Describe how a vocabulary converts tokens to indices and handles tokens absent from its mapping.
  Why it matters: A vocabulary makes tokenized text numeric while allowing inputs to contain unfamiliar terms.
• Standardization benefits and limits: Explain why standardization can make similar text more alike and why modern models often use it lightly.
  Why it matters: Normalization can reduce needless variation, but overdoing it can erase information useful for understanding text.

Key terms: standardization, splitting, indexing, tokenization, tokenizer, tokens, regular expressions, character tokens, word tokens, vocabulary

14.2.1 Character and word tokenization

This section builds simple character- and word-level tokenizers and explains the tradeoff between vocabulary size and encoded sequence length.

• Character and word tokenizers: Describe how the simple character-level tokenizer maps lowercased input characters to integers, and how the word-level tokenizer uses the same approach with a different splitting step.
  Why it matters: The splitting step determines what each integer in the encoded text represents.
• Vocabulary limits and unknown tokens: Explain why the vocabulary can be limited to the most common tokens and what is lost when a word-level vocabulary is restricted too aggressively.
  Why it matters: A vocabulary limit controls model size, but setting it too low can erase useful distinctions in the text.
• Vocabulary and sequence tradeoff: Contrast the vocabulary sizes and output lengths of character- and word-level tokenization, and explain why shorter word-token sequences matter for model input.
  Why it matters: This tradeoff explains why neither character nor word tokenization is an effortless choice for large-scale text.

Key terms: character-level tokenizer, word-level tokenizer, standardization, splitting step, vocabulary, most common tokens, parameters, [UNK] token, sequence, compression

14.2.2 Subword tokenization

This section explains how subword tokenization uses byte-pair encoding to balance short token sequences with coverage of rare words, and how to choose among tokenization approaches.

• Tokenization as compression tradeoff: Explain why subword tokenization seeks both short token sequences and a small vocabulary, combining advantages of word- and character-level tokenization.
  Why it matters: This tradeoff explains why subwords are useful rather than simply choosing whole words or individual characters.
• Byte-pair vocabulary construction: Trace how byte-pair encoding builds a subword vocabulary from characters by repeatedly merging the most frequent adjacent symbol pair.
  Why it matters: The repeated merges explain how useful subwords emerge from the text rather than being specified by hand.
• Common words and rare-word coverage: Explain why learned subwords can keep common inputs relatively short while representing rare words without the unknown-word token, and describe the limited fallback condition.
  Why it matters: This is the practical payoff of retaining characters while learning larger pieces for frequent text.
• Tokenization choice by use case: Distinguish the tokenization choice for a pretrained model from the choices available when building a model from scratch, including when subword, character, or word tokens may be useful.
  Why it matters: The best tokenizer depends on whether learned model weights must be reused and what information the task needs.

Key terms: subword tokenization, WordTokenizer, CharTokenizer, vocabulary, byte-pair encoding, merge, symbol pair, subword vocabulary, merge rules, SubWordTokenizer

14.3 Sets vs. sequences

This section explains why word order complicates text representation and contrasts bag-of-words and sequence-model approaches to handling it.

• Categorical token representation: Explain that individual tokens can be treated as categorical features and encoded as feature-space dimensions or token vectors.
  Why it matters: This separates the familiar task of encoding tokens from the harder task of representing their order.
• Word order and meaning: Explain why word order matters to meaning but is not a simple, universal ordering like steps in a timeseries.
  Why it matters: The difficulty of representing order motivates the different text-modeling approaches in this section.
• Set and sequence approaches: Distinguish bag-of-words models, recurrent models, and Transformers by how they handle word order.
  Why it matters: How a model handles order is the central distinction between the text-classification approaches introduced here.

Key terms: categorical features, feature space, category vectors, token vectors, word order, canonical order, meaning, ambiguity, bag-of-words models, recurrent models

14.3.1 Loading the IMDb classification dataset

This section inspects the IMDb review files, sets aside validation data, and loads training, validation, and test datasets for text classification.

• IMDb review directory labels: Describe how the IMDb review directories distinguish positive, negative, and unlabeled reviews.
  Why it matters: The directory layout tells you which reviews have labels available for classification.
• Review inspection before modeling: Explain why the section looks at the contents of review files before modeling.
  Why it matters: Seeing real reviews gives you a concrete sense of the inputs the model will handle.
• Validation split and dataset loading: Trace how the section prepares the IMDb files for training: leave out unsup/, split validation reviews from the training files, and load separate training, validation, and test Dataset objects.
  Why it matters: This preparation gives the model labeled training data and a separate set for monitoring accuracy.

Key terms: train, test, pos/, neg/, unsup/, inspect, text files, modeling, validation set, text_dataset_from_directory

14.4 Set models

This section introduces bag-of-words text classification by discarding token order, learning weights for word presence, and encoding reviews as multi-hot vectors.

• Unordered bag-of-words representation: Explain how a review becomes an unordered set of tokens and how a bag-of-words model uses the presence of individual words to learn weights associated with review sentiment.
  Why it matters: It explains what information this simple text classifier keeps—and what it throws away.
• Multi-hot review encoding: Describe how a set of token IDs becomes a multi-hot vector and why that vector can serve as model input.
  Why it matters: It connects the unordered set of words to a numerical input the model can train on.

Key terms: set, bag-of-words model, token IDs, multi-hot encoding, fixed-sized vector, vocabulary

14.4.1 Training a bag-of-words model

The section builds and evaluates a word-level bag-of-words classifier for IMDb reviews, then explains how to keep text preprocessing efficient during training.

• Word-level multi-hot encoding: Describe how TextVectorization learns a vocabulary from the reviews and turns each review into a vector indicating which vocabulary words are present.
  Why it matters: This encoding is the input representation used by the section’s classifier.
• Simple classifier and results: Explain how simple the linear bag-of-words model is and summarize its training and test results.
  Why it matters: The result establishes how far a small, inexpensive bag-of-words model can go on this task.
• Word tokenization choice: Justify the use of word tokenization rather than character tokenization in this example, and describe why subword tokenization is not needed here.
  Why it matters: The tokenization choice determines what information the bag-of-words model can use.
• Efficient on-the-fly preprocessing: Explain why text tokenization can bottleneck accelerator training and how the section uses tf.data to keep batches ready.
  Why it matters: Keeping preprocessing fast enough prevents costly accelerators from sitting idle.

Key terms: TextVectorization, adapt(), vocabulary, multi-hot encoding, bag-of-words, linear model, bias term, validation performance, test set, EarlyStopping

14.4.2 Training a bigram model

The section adds bigrams to a set-based IMDb sentiment classifier, shows the accuracy gain, and explains why longer n-grams become impractical.

• Local word order and bigrams: Explain why a unigram bag-of-words model can miss meaning and how bigrams capture some local word order.
  Why it matters: Bigrams let a simple text classifier distinguish some phrases that isolated words cannot.
• Bigram vocabulary and encoding: Describe how the section adds bigrams to the classifier’s input without changing its linear model.
  Why it matters: The improvement comes from changing the text features, not from introducing a new classifier architecture.
• Accuracy gain and n-gram limits: State the reported result of adding bigrams and explain why extending the same approach to much longer n-grams is problematic.
  Why it matters: The result shows both the value of local order and the limit of representing longer phrases as vocabulary entries.

Key terms: bigram, unigram, n-gram, bag-of-words, TextVectorization, vocabulary, adapt(), multi-hot encode, unigrams, bigrams

14.5 Sequence models

This section motivates sequence models for text classification and explains how to prepare variable-length reviews as fixed-length batches of token IDs.

• Learning from word order: Explain why sequence models receive an ordered token sequence instead of relying only on manually engineered word-order features.
  Why it matters: This explains what sequence models are intended to gain over the earlier text classifiers.
• Fixed-length token sequence batches: Describe how tokenized reviews of different lengths are made suitable for batch computation, including the section's IMDb length choice.
  Why it matters: Equal-length sequences make it practical to process many reviews together efficiently.

Key terms: sequence models, raw word sequence, positional dependencies, token IDs, [PAD], truncation, padding, (batch_size, sequence_length)

14.5.1 Training a recurrent model

This section builds a bidirectional LSTM text classifier using one-hot token vectors, then explains why its large inputs make it slower and less accurate than the earlier set-based models.

• One-hot token representation: Explain how token IDs are converted into input for the LSTM model: each token becomes a floating-point vector with one 1 at its vocabulary index and 0s elsewhere.
  Why it matters: This representation determines the size of the data the LSTM must process.
• Bidirectional LSTM classification: Describe why the sequence model uses a bidirectional LSTM for classification: information can propagate both forward and backward along the token sequence.
  Why it matters: It explains the model choice without requiring the review to be processed in only one direction.
• One-hot LSTM tradeoff: Explain why this working LSTM classifier trains slowly and why the section judges one-hot encoding a poor choice compared with the earlier set-based models.
  Why it matters: The result motivates looking for a better way to represent words.

Key terms: token ID, one-hot, vocabulary, multi-hot encoding, bidirectional LSTM, token sequence, classification, one-hot encoding, set-based models, word embeddings

14.5.2 Understanding word embeddings

This section explains why word embeddings represent words as dense, learned vectors whose geometry can reflect meaning, unlike one-hot vectors.

• One-hot independence assumption: Explain why one-hot word vectors cannot express similarity between words and why that is a poor assumption for language.
  Why it matters: It explains why a representation that gives every word a separate slot misses useful relationships between words.
• Dense semantic word vectors: Describe how word embeddings differ from one-hot vectors in density and dimensionality, and how closeness between embedding vectors can reflect related meanings.
  Why it matters: It connects the compact form of embeddings to their ability to represent relationships between words.
• Meaningful embedding directions: Explain how a direction in an embedding space can represent a recurring semantic relationship, using an example from the section.
  Why it matters: It shows that embedding geometry can capture relationships between pairs of words, not just similarity between individual words.

Key terms: one-hot encoding, orthogonal, feature engineering, word embeddings, dense vectors, sparse vectors, embedding space, semantic distance, geometric transformations, gender vectors

14.5.3 Using a word embedding

This section explains how Keras learns task-specific word embeddings, uses padding masks with an LSTM, and why the resulting efficiency gains do not necessarily improve accuracy.

• Task-specific embedding spaces: Explain why a useful word-embedding space depends on the task and can be learned as part of training a model.
  Why it matters: It explains why the embedding is trained for the classification problem rather than treated as a universal word representation.
• Embedding lookup and learned weights: Describe how a Keras Embedding layer turns integer token sequences into vectors, and how those vectors change during training.
  Why it matters: The lookup and learned weights explain how integer text inputs become useful inputs to the LSTM.
• Padding masks for LSTM outputs: Trace why padded tokens could distort an LSTM’s final representation and how mask_zero=True prevents that.
  Why it matters: Masking keeps padding added for sequence processing from determining the classifier’s final representation.
• Embedding efficiency and accuracy limit: Distinguish the embedding model’s efficiency gains from its unchanged classification accuracy, and explain the section’s account of that result.
  Why it matters: A smaller, faster model is not automatically a more accurate one; the training data can still limit its learned representations.
• Text augmentation trade-offs: Explain what simple text augmentation can help with and why the section generally favors obtaining more text examples.
  Why it matters: It cautions against treating text edits as reliably label-preserving ways to expand a small dataset.

Key terms: word-embedding space, semantic relationships, Embedding layer, input_dim, output_dim, backpropagation, embedding_size, mask_zero=True, [PAD], mask

14.5.4 Pretraining a word embedding

The section explains why to pretrain word embeddings, then builds a Continuous Bag of Words task from IMDb reviews to learn embeddings for an LSTM classifier.

• Purpose of embedding pretraining: Explain why an unsupervised pretraining task can help with text classification when labeled examples are limited, and how it primes word embeddings before the final task.
  Why it matters: This explains why the extra IMDb reviews are useful even without classification labels.
• CBOW context examples: Describe how the section turns IMDb review text into Continuous Bag of Words training examples, including which token is the label and which tokens form its context.
  Why it matters: The window creates a prediction task from reviews without needing human-provided labels.
• Context prediction embedding model: Trace how the simple CBOW model uses context-token embeddings to predict the missing middle token, and explain what repeated training improves.
  Why it matters: The useful output of this prediction exercise is the learned word embedding.
• Caching preprocessed training data: Justify caching the batched CBOW dataset when preprocessing is slower than training the simple model.
  Why it matters: It shows why faster model computation does not necessarily mean faster training epochs.

Key terms: pretraining, unsupervised task, word embedding, Continuous Bag of Words (CBOW), sliding window, unordered context, TextVectorization, Embedding layer, GlobalAveragePooling1D, embedding space

14.5.5 Using the pretrained embedding for classification

The section applies a pretrained CBOW word embedding to an LSTM classifier, evaluates the result, and explains when a simpler set-based classifier may be preferable.

• Pretrained embedding transfer: Describe how the CBOW embedding is reused in the LSTM classifier and what role its weights play.
  Why it matters: The pretrained weights give the classifier a better starting point without changing its normal training process.
• Performance and data constraint: Explain the pretrained LSTM’s performance relative to the set-based models and why using word order has not yet produced a clear advantage.
  Why it matters: The result does not show that word order is useless; it shows that exploiting it requires enough ordered training data.
• Set-based model tradeoff: Justify when a simpler set-based classifier may be a cost-effective choice despite its lower potential accuracy.
  Why it matters: Choosing a text classifier involves weighing speed and compute cost against accuracy.

Key terms: CBOW embedding layer, LSTM embedding layer, embedding weights, initializer, LSTM, set-based models, unigram model, bigram model, ordered data, data-constrained

14.6 Summary