How deeply does Sutskever's List cover TPU?

Defines the TPU and gives per-device capacity, but does not describe its internal design.

TPU

A tensor processing unit is a specialized hardware accelerator designed to run and train artificial neural networks.

A tensor processing unit (TPU) is Google’s custom-built accelerator for running and training artificial neural networks. In GPipe experiments, TPU hardware provided the platform for scaling Transformer training across multiple partitions. section 6.3.2

Scaling a Transformer

A single 16 GB TPU can store approximately 282 million parameters. With GPipe, researchers trained an 83.9-billion-parameter Transformer across 128 TPU partitions—nearly 300 times the size achievable on one device. section 6.3.2

How Sutskever's List covers it

liveBook manning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Starting Data Analytics with Generative AI and PythonA TPU is a specialized accelerator designed for high-performance machine-learning computations, especially neural-network operations.
liveBook manning.com

Deep Learning with Python, Third EditionA specialized tensor-processing chip and cloud-accessible hardware accelerator used to speed up machine learning training.
liveBook manning.com

Graph Neural Networks in ActionA tensor processing unit is a specialized hardware accelerator designed for machine-learning computations involving tensor operations.
liveBook manning.com

Deep Learning with JAXA tensor processing unit hardware backend used to accelerate machine-learning computations.

Compare how 5 books explain TPU →

Related concepts

JAXused by · JAX is a Python library for high-performance array-based numerical computing with composable transformations including automatic differentiation, compilation, vectorization, and parallelization.
DistributedDataParallelused by · DistributedDataParallel is a PyTorch module that replicates a model across processes or devices and synchronizes gradients during data-parallel training.
language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.