10 Applications of Transformers: hands-on with BERT
This chapter covers
- Creating a BERT layer for importing existing BERT models
- Training BERT on data
- Fine-tuning BERT
- Extracting embeddings from BERT, and inspecting them
The chapter addresses the practicalities of working with the Transformer BERT in your implementations. We will not implement BERT ourselves. That would be a daunting job, and quite unnecessary, since BERT has been implemented efficiently in a variety of frameworks, including Keras. But we will get close to the inner workings of BERT code. We saw in Chapter 9 that BERT has been reported to improve NLP applications significantly. While we do not carry out an extensive comparison ourselves in this chapter, you are encouraged to revert back to the applications in the previous chapters, and swap for instance Word2Vec embeddings with BERT embeddings. With the material of Chapter 9 and 10 combined, you should be able to do so.
The financial costs of pretraining BERT and related models like XLNET from scratch on large amounts of data can be prohibitive. The original BERT paper ([Devlin2018-10], see Chapter 9) mentions that
- "[The] training of BERT – Large was performed on 16 Cloud TPUs (64 TPU chips total) [with several pretraining phases]. Each pretraining [phase] took 4 days to complete.”