3 Getting started with baselines: Benchmarking and optimization

In this chapter, we continue our direct dive into solving NLP problems we started in the previous chapter. We continue with our goal to establish a set of baselines for a pair of concrete NLP problems, which we will later be able to use to measure progressive improvements gained from leveraging increasingly sophisticated transfer learning approaches. We complete the exercise that we started in chapter 2, where we introduced a pair of practical problems, preprocessed the corresponding data, and initiated baselining by exploring some generalized linear methods. In particular, we introduced the email spam and the Internet Movie Database (IMDB) movie review classification examples and used logistic regression and a support vector machine (SVM) to baseline them.

3.1 Decision-tree-based models

3.1.1 Random forests (RFs)

3.1.2 Gradient-boosting machines (GBMs)

3.2 Neural network models

3.2.1 Embeddings from Language Models (ELMo)

3.2.2 Bidirectional Encoder Representations from Transformers (BERT)

3.3 Optimizing performance

3.3.1 Manual hyperparameter tuning

3.3.2 Systematic hyperparameter tuning

Summary