Overfitting occurs when a model keys too closely off peculiarities in its training data instead of learning patterns that transfer to new data. In a toxicity filter, a large vocabulary, few labeled examples, and class imbalance can make the model depend on a handful of toxic words; users can evade it with unseen synonyms. section 4.4
Recognize and reduce overfitting by comparing performance and controlling model capacity:
- Watch for training accuracy that keeps rising while test accuracy stalls or falls; a CNN reached 94% training accuracy but only 78% test accuracy after extended training. section 7.4.2
- Reduce unnecessary capacity by limiting parameters, channels, layers, or output dimensionality; these choices can reduce overfitting but may slow convergence. section 7.4.2 section 7.5.1
- Use lower-dimensional topic representations, pretrained models, or dropout to encourage generalization when labeled language data is sparse. section 4.4 section 7.7
- Experiment systematically and document hyperparameters, training duration, dropout, layer counts, losses, and test results. section 8.4.4
A CNN learns the training set too well
A disaster-post classifier reached 79% test accuracy after seven epochs while keeping overfitting low. Continuing to epoch 20 raised training accuracy above 90% but left test accuracy near 78%. Inspecting layer parameters and reducing model capacity can help identify the cause; dropout or a pretrained model provides additional countermeasures. section 7.4.2
import pandas as pd
import jsonlines
with jsonlines.open('experiments.jsonl') as fin:
lines = list(fin)
df = pd.DataFrame(lines)
df.to_csv('experiments.csv')
cols = 'rnn_type epochs lr num_layers dropout epoch_time test_loss'
cols = cols.split()
df[cols].round(2).sort_values('test_loss').head(10)epochs lr num_layers dropout epoch_time test_loss
37 12 2.0 2 0.2 35.43 5.23
28 12 2.0 1 0.0 22.66 5.23
49 32 0.5 2 0.0 32.35 5.22
57 32 0.5 2 0.2 35.50 5.22
38 12 2.0 3 0.2 46.14 5.21
50 32 0.5 3 0.0 37.36 5.20
52 32 2.0 1 0.0 22.90 5.10
55 32 2.0 5 0.0 56.23 5.09
53 32 2.0 2 0.0 32.49 5.06
54 32 2.0 3 0.0 38.78 5.04
Compare recorded RNN experiments by epochs, learning rate, layer count, dropout, training time, and test loss when tuning for generalization. section 8.4.4
