Self-supervised learning creates training labels from the data rather than requiring people to label every example. It addresses a practical obstacle in deep learning: useful models can require far more labeled examples than people can feasibly provide. In outlier detection, its promise is to learn representations that capture subtle relationships in large datasets. section 16.5 section 16.0
Learn from image rotations
Start with 100,000 unlabeled images and rotate each by 0, 90, 180, and 270 degrees. This produces 400,000 images whose rotation labels are known automatically. Train a neural network to predict the rotation. Rotation prediction is a pretext task: the intended product is not the prediction itself, but an embedding from an intermediate layer that may be useful for another task. section 16.5.1
Figure 16.9 Neural network to predict the rotation of images. The input is likely the pixels of the image though may be embeddings representing the images created by another neural network. The output neurons predict the four types of rotation used in this example. Any number of rotations, or other modifications of the images, may also be used. The second-last layer may be used as embeddings for the images.
The rotation-prediction network illustrates the distinction between its four-class training output and the second-last layer that can provide image embeddings. section 16.5.1
Representations learned on one task can sometimes support another model through transfer learning and fine-tuning. This can reduce subsequent training because the first network has already learned relevant features. Transfer is less straightforward for tables: unlike images or text, different tables may have little in common. section 16.5.3
For tabular outlier detection, the central challenge is choosing a pretext task that yields useful row embeddings. Approaches under investigation include:
- Predicting transformations, such as arithmetic changes to numeric fields or shuffled cells. section 16.5.4
- Contrastive learning and masked learning. section 16.5.4
A transformation that preserves an image's essential content may change a table row's meaning when even one cell changes. Self-supervised methods therefore look promising for images and other modalities but remain difficult to apply effectively to tables. Very large tables with many features or complex feature relationships may be exceptions; the method should not be assumed to outperform established tabular detectors. section 16.5.4 section 16.6
