10 Natural Language Processing with TensorFlow: Language Modelling
This chapter covers,
- Implementing a TensorFlow data pipeline that can generate inputs and targets from free text for a language modelling task
- Implementing a GRU based language model
- Defining and perplexity metric in TensorFlow and understand how to interpret it
- Training the language model on a text corpus
- Defining an inference model to generate text based on the trained GRU model
- Implementing beam-search to uplift the quality of generated text
In the last chapter, we discussed an important NLP task called sentiment analysis. Here we will discuss a new task known as language modelling. Language modelling has been at the heart of natural language processing. Language modelling refers to the task where we predict the next word given a sequence of previous words. For example, given the sentence, I went swimming in the ____, the model would predict the word “pool”. Ground-shattering models like BERT (Bidirectional Encoder Representation from Transformers is a type of transformer based model) are trained using language modelling tasks. This is a prime example of how language modelling can help to actualize innovative models that goes on to be used in a plethora of areas and use cases.