concept t - SNE in category machine learning

This is an excerpt from Manning's book Ensemble Methods for Machine Learning MEAP V01. Login to get full access to this book.

We’ll use a nonlinear dimensionality reduction technique known as t-distributed stochastic neighbor embedding or t-SNE. t-SNE is a highly effective pre-processing technique for the digits data set and extracts an effective embedding in a two-dimensional space.

Figure 4.14 Visualization of the 2d embedding of digits data set produced by t-SNE, which is able to embed and separate the digits, effectively clustering them.

This is an excerpt from Manning's book Machine Learning with R, the tidyverse, and mlr. Login to get full access to this book.

t-SNE is one of the most popular nonlinear dimension-reduction algorithms. It measures the distance between each observation in the dataset and every other observation, and then randomizes the observations across (usually) two new axes. The observations are then iteratively shuffled around these new axes until their distances to each other in this two-dimensional space are as similar to the distances in the original high-dimensional space as possible.

UMAP is another nonlinear dimension-reduction algorithm that overcomes some of the limitations of t-SNE. It works similarly to t-SNE (finds distances in a feature space with many variables and then tries to reproduce these distances in low-dimensional space), but differs in the way it measures distances.

After converting the distances to probabilities, the probabilities for each case are scaled by dividing them by their sum. This makes the probabilities sum to 1 for every case in the dataset. Using different standard deviations for different densities, and then normalizing the probabilities to 1 for every case, means if there are dense clusters and sparse clusters of cases in the dataset, t-SNE will expand the dense clusters and compress the sparse ones so they can be visualized more easily together. The exact relationship between data density and the standard deviation of the normal distribution depends on a hyperparameter called perplexity, which we’ll discuss shortly.

Figure 14.1. t-SNE measures the distance from each case to every other case, converted into a probability by fitting a normal distribution over the current case. These probabilities are scaled by dividing them by their sum, so that they add to 1.

The weaknesses of t-SNE and UMAP are these:

The new axes of t-SNE and UMAP are not directly interpretable in terms of the original variables.

t-SNE cannot make predictions on new data (different result each time).

t-SNE is computationally expensive.

t-SNE doesn’t necessarily preserve global structure.