10 Part-of-speech tagging and word-sense disambiguation

 

This chapter covers

  • Disambiguating language by predicting nouns, verbs, and adjectives from past data
  • Making decisions and explaining them using hidden Markov models (HMMs)
  • Using TensorFlow to model explainable problems and collect evidence
  • Computing HMM initial, transition, and emission probabilities from existing data
  • Creating a part-of-speech (PoS) tagger from your own data and larger corpora

You use language every day to communicate with others, and if you are like me, sometimes you scratch your head, especially if you are using the English language. English is known to have a ton of exceptions that make it difficult to teach non-native speakers, along with your little ones who are growing up trying to learn it themselves. Context matters. Conversationally, you can use tools such as hand motions, facial expressions, and long pauses to convey additional context or meaning, but when you are reading language as written text, much of that context is missing, and there is a lot of ambiguity. Parts of speech (PoS) can help fill that missing context to disambiguate words and make sense of them in text. PoS tells you whether the word is being used is an action word (verb), whether it refers to an object (noun), whether it describes a noun (adjective), and so on.

10.1 Review of HMM example: Rainy or Sunny

10.2 PoS tagging

10.2.1 The big picture: Training and predicting PoS with HMMs

10.2.2 Generating the ambiguity PoS tagged dataset

10.3 Algorithms for building the HMM for PoS disambiguation

10.3.1 Generating the emission probabilities

10.4 Running the HMM and evaluating its output

10.5 Getting more training data from the Brown Corpus

10.6 Defining error bars and metrics for PoS tagging

Summary