AmoebaNet is a convolutional neural network used to evaluate GPipe, a method for training models across multiple devices. On one 8-GB GPU, training was limited to about 82 million parameters; partitioning the model across eight GPUs raised the limit more than twentyfold, to 1.8 billion. Despite this memory gain, convolutional networks scaled less cleanly than Transformers. section 6.3.2
AmoebaNet
AmoebaNet is a convolutional neural network architecture discovered through neural architecture search for image recognition.
How Sutskever's List covers it
- How deep it goes
- Compares parameter capacity across one and eight GPUs and notes that convolutional networks scale less cleanly than Transformers.
- Examples
- Contrasts an 82-million-parameter single-GPU limit with a 1.8-billion-parameter model partitioned across eight GPUs.
- What you can do after
- Evaluate how distributing a convolutional network across GPUs can increase its trainable size while scaling less cleanly than a Transformer.
- How it connects
- Connects AmoebaNet to GPipe, pipeline parallelism, model partitioning, rematerialization, and convolutional neural networks.
Related concepts
- AmoebaNet-Bincludes · AmoebaNet-B is a convolutional neural network architecture in the AmoebaNet family for visual recognition.
- GPipeused by · GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.