liveBook · Manning

Alpa

As explained in Deep Learning with JAX

A system for automatically parallelizing and training or serving large-scale neural networks on distributed clusters.

Need to train or serve a neural network too large for one device? Alpa automatically parallelizes single-device code across distributed clusters, combining data, operator, and pipeline parallelism. section 12.1.2

Alpa is positioned for large-model workflows such as:

  • Training or fine-tuning vision transformers for image classification. section 12.1.2
  • Fine-tuning OPT language models on AWS p3.16xlarge instances with eight 16GB V100 GPUs. section 12.1.2
  • Running GPT-2-like training examples alongside broader large-scale neural-network serving. section 12.1.2

How Deep Learning with JAX covers it

How deep it goes
Defines automatic data, operator, and pipeline parallelization, but does not explain its algorithms or implementation internals.
Examples
Uses vision-transformer classification, OPT fine-tuning on eight V100 GPUs, and a GPT-2-like example.
What you can do after
Operate distributed training or serving workflows from single-device neural-network code.
How it connects
Connects single-device code with data parallelism, operator parallelism, pipeline parallelism, vision transformers, and language models.
Cover of Deep Learning with JAX
liveBookmanning.com
Read Deep Learning with JAX on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with JAX. The section links point to the parts of the book it draws on.