Dropout is a neural-network regularization technique that randomly hides neuron outputs during training. By silencing some pathways, it prevents a model from relying too heavily on particular words, patterns, or filters. section 7.4.4
In a text CNN, place dropout near the end, before the fully connected prediction layer. Hiding pooled pattern features forces the classifier to diversify its learned evidence and improves robustness to unusual language. section 7.4.4
Tune dropout as a practical hyperparameter:
- Try ratios around 30%–50% for many neural networks, while keeping single-layer CNN ratios at or below about 20%. section 7.5.5
- Increase dropout when the model overfits and simplifying the architecture is not suitable. section 8.4.4
- Avoid setting the percentage much above 50%: learning can slow and validation error may become unstable. section 8.4.4
- Record each experiment with its architecture, learning rate, dropout, training time, and test loss before choosing the next adjustment. section 8.4.4
import pandas as pd
import jsonlines
with jsonlines.open('experiments.jsonl') as fin:
lines = list(fin)
df = pd.DataFrame(lines)
df.to_csv('experiments.csv')
cols = 'rnn_type epochs lr num_layers dropout epoch_time test_loss'
cols = cols.split()
df[cols].round(2).sort_values('test_loss').head(10)epochs lr num_layers dropout epoch_time test_loss
37 12 2.0 2 0.2 35.43 5.23
28 12 2.0 1 0.0 22.66 5.23
49 32 0.5 2 0.0 32.35 5.22
57 32 0.5 2 0.2 35.50 5.22
38 12 2.0 3 0.2 46.14 5.21
50 32 0.5 3 0.0 37.36 5.20
52 32 2.0 1 0.0 22.90 5.10
55 32 2.0 5 0.0 56.23 5.09
53 32 2.0 2 0.0 32.49 5.06
54 32 2.0 3 0.0 38.78 5.04
Compare RNN experiments by dropout, layer count, learning rate, training time, and test loss. section 8.4.4
