liveBook · Manning

iGPT

As explained in Sutskever's List

A GPT-style transformer trained without labels to predict the next pixel in low-resolution images and whose representations can support downstream classification.

iGPT is a GPT-style transformer trained without labels to predict the next pixel in low-resolution images. By modeling image structure rather than class labels, it learns representations that support later supervised classification. section 8.3

A surprising classification result

On CIFAR-10, iGPT reaches 96.3%, exceeding ResNet-152 despite never being trained to classify images. Its strongest classification features appear in the middle layers; accuracy falls by about 2.4% at the final layer. section 8.3

The result illustrates three linked ideas:

  • Next-pixel prediction connects compression, prediction, and model inference. section 8.3
  • Modeling P(X) preserves semantic structure alongside low-level visual detail, unlike objectives focused directly on P(Y|X). section 8.3

How Sutskever's List covers it

How deep it goes
Explains the training objective, representation behavior, and layerwise classification trade-off without detailing transformer internals.
Examples
Uses low-resolution image prediction, CIFAR-10 classification, ResNet-152 comparison, and middle-layer feature analysis.
What it assumes
Assumes familiarity with neural networks, supervised classification, and basic predictive modeling.
What you can do after
Evaluate how an unlabeled predictive objective can produce transferable classification representations.
How it connects
Connects minimum description length, predictive modeling, supervised learning, and ResNet architectures.
Cover of Sutskever's List
liveBookmanning.com
Read Sutskever's List on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.