An unsupervised task learns useful model parameters without labeled targets by deriving supervision from the data itself. In text modeling, it turns abundant unlabeled text into a way to prime a model before specialization. section 14.5.4 section 14.6
Predicting a missing word with CBOW
A Continuous Bag of Words task slides a window across text, uses surrounding words such as “sail,” “wave,” and “mast” as context, and predicts the missing middle word, perhaps “boat” or “ocean.” For IMDb, the task reuses the text vocabulary and learns a 64-dimensional vector for each word. Context windows are prepared in TensorFlow, an embedding and classifier are built with Keras, and the model is trained on batches of unlabeled reviews. section 14.5.4
Use an unsupervised task when:
- Labeled examples are scarce but large amounts of related or arbitrary text are available. section 14.5.4
- A sequence model needs general language patterns before learning a downstream classification problem. section 14.6
- A simple predictive objective can convert context into supervision, as in CBOW’s missing-word prediction. section 14.5.4
