A pretext task is an automatically labeled training task whose target is not the desired prediction. Solving it forces a neural network to learn representations that can transfer to other machine-learning tasks. section 16.5.1
The task starts with abundant unlabeled data. A self-supervised transformation supplies targets without hand-labeling, and a network learns embeddings by predicting those targets. section 16.5.1
Predicting image rotations
Rotate each image by 0, 90, 180, or 270 degrees, label the applied rotation automatically, and train a four-class classifier. The second-last layer can provide embeddings for later classification or regression. section 16.5.1
Useful pretext-task patterns include:
- Predict transformations such as rotations, crops, stretches, discoloration, numeric operations, or reordered cells. section 16.5.4
- Shuffle scaled table cells and predict their correct order, encouraging the network to learn relationships among features. section 16.5.4
- Use contrastive learning or masked learning when transformation-based targets are difficult to design. section 16.5.4
Designing pretext tasks is easier for images than tables. Small changes to a table cell may change its meaning, and tabular data lacks universally reliable transformations that preserve semantic content. section 16.5.4
A network trained on a pretext task can act as a foundation model: another model may reuse its embeddings and fine-tune only the final or added layers. This can reduce later training effort, although transfer is much harder between unrelated tables. section 16.5.3
