Hyperparameter tuning is the systematic adjustment of function arguments or configuration values to improve a model or processing pipeline. Useful choices depend on the task: changing an NLP vectorizer from single tokens to 2-grams can improve retrieval, while neural-network tuning balances accuracy, overfitting, and training time. section 3.4 section 7.5.5
A practical tuning loop includes:
- Choose parameters to vary, such as n-gram range, learning rate, layer count, hidden units, batch size, sequence length, or dropout. section 3.4 section 8.4.4
- Record every experiment and compare a meaningful evaluation result, especially test or validation accuracy and loss. section 7.5.5 section 8.4.4
- Adjust deliberately rather than maximizing model size; simpler networks can train faster and sometimes perform better. section 8.4.4
- Use dropout to address overfitting, while recognizing that excessive dropout can slow learning and destabilize validation error. section 7.5.5 section 8.4.4
Comparing CNN configurations
A recorded CNN search compares kernel sizes, learning rates, sequence lengths, vocabulary size, bidirectional sensitivity, and dropout. One listed configuration—kernel sizes [2, 3, 4, 5], learning rate 0.001, sequence length 32, sensitivity enabled, vocabulary 2,000, and dropout 0.2—reaches 87.27% training accuracy and 79.00% test accuracy. section 7.5.5
import pandas as pd
import jsonlines
with jsonlines.open('experiments.jsonl') as fin:
lines = list(fin)
df = pd.DataFrame(lines)
df.to_csv('experiments.csv')
cols = 'rnn_type epochs lr num_layers dropout epoch_time test_loss'
cols = cols.split()
df[cols].round(2).sort_values('test_loss').head(10)epochs lr num_layers dropout epoch_time test_loss
37 12 2.0 2 0.2 35.43 5.23
28 12 2.0 1 0.0 22.66 5.23
49 32 0.5 2 0.0 32.35 5.22
57 32 0.5 2 0.2 35.50 5.22
38 12 2.0 3 0.2 46.14 5.21
50 32 0.5 3 0.0 37.36 5.20
52 32 2.0 1 0.0 22.90 5.10
55 32 2.0 5 0.0 56.23 5.09
53 32 2.0 2 0.0 32.49 5.06
54 32 2.0 3 0.0 38.78 5.04
Inspect, convert, sort, and compare recorded RNN experiments by epochs, learning rate, layer count, dropout, training time, and test loss. section 8.4.4
