A hyperparameter is a setting you choose to control an NLP pipeline or model, rather than a value the model learns from training data. Changing one can alter what the system represents, how it trains, or how well it performs. Hyperparameter tuning means trying settings and checking which ones work for the task. section 3.4 section 4.5.1
Change what a search system can match
Suppose a question asks, “What is algorithmic bias?” Counting individual words gives one sentence the highest cosine similarity, even though another may be a better definition. Setting CountVectorizer’s ngram_range to (1, 2) also counts adjacent word pairs, including “algorithmic bias.” That may improve matching, but the resulting vectors are longer: this corpus grows from 240 to 616 dimensions. More features are not a free improvement. section 3.4
>>> ngram_vectorizer = CountVectorizer(ngram_range=(1, 2))
>>> ngram_vectors = ngram_vectorizer.fit_transform(corpus)
>>> ngram_vectors
<16x616 sparse matrix of type '<class 'numpy.int64'>'
with 772 stored elements in Compressed Sparse Row format>
The CountVectorizer call shows where the n-gram choice is set and reports the resulting vector shape. section 3.4
Other choices depend on the model. An LDiA topic model needs a topic count before it can assign words to topics; you can adjust that count and test whether the resulting representation helps a task such as classification. A neural-network optimizer needs settings such as learning rate and momentum. Learning rate controls how far weights move in response to error: too large a step can overshoot a minimum. section 4.5.1 section 5.2.5
A practical tuning loop is:
- Choose a setting to vary and a relevant measure of success, such as test accuracy or performance on labeled documents. section 4.5.1 section 7.5.5
- Run alternatives and compare their results. CNN experiments vary kernel sizes, learning rate, sequence length, case sensitivity, vocabulary size, and dropout; their training and test accuracies differ. section 7.5.5
- Keep the dataset, software, and random seed with a promising result so you can reproduce it. Watch for a widening training–test gap rather than trusting training accuracy alone. section 7.5.5
Listing 7.31 CNN hyperparameter tuning
learning seq case vocab training test
kernel_sizes rate len sens size dropout accuracy accuracy
[2] 0.0010 32 False 2000 NaN 0.5790 0.5459
[1 2 3 4 5 6] 0.0010 40 False 2000 NaN 0.7919 0.7100
[2 3 4 5] 0.0015 40 False 2000 NaN 0.8038 0.7152
[1 2 3 4 5 6] 0.0010 40 True 2000 NaN 0.7685 0.7520
[2] 0.0010 32 True 2000 0.2 0.8472 0.7533
[2 3 4 5] 0.0010 32 True 2000 0.2 0.8727 0.7900
The CNN results table makes the trade-off visible: combinations with higher training accuracy do not necessarily have the highest test accuracy. section 7.5.5
Store configuration alongside what gives it meaning. An RNN constructor can retain its hidden-vector size, vocabulary mapping, and category labels so a saved model uses consistent inputs and readable outputs when reloaded. Interactive experiments with a surname classifier also let you vary sample count and learning rate, then inspect validation accuracy—not just plausible-looking predictions. section 8.2.1 section 8.2.2
