liveBook · Manning

Accelerator

As explained in Machines that Think

A hardware accelerator is a specialized processor designed to execute particular computational workloads more efficiently than general-purpose processors.

An accelerator is a processor designed to execute particular workloads efficiently. In large language models, accelerators handle the repeated, large-scale matrix operations involved in training and the computation needed to produce responses. section 4.6.1 section 8.1.2

Training repeatedly updates model parameters through forward and backward passes, gradient computations, and optimization. Inference uses frozen parameters and typically requires only a forward pass. It is less expensive than training, though it can still be intensive at scale. section 4.6.1

Accelerator designs make different trade-offs:

  • GPUs perform many operations in parallel. Their flexibility and mature software ecosystem have made them standard for training and running modern AI models, though highly specialized workloads can expose efficiency limits. section 8.1.2
  • TPUs and custom accelerators such as AWS Trainium and AWS Inferentia optimize computation and data flow for machine learning. They can outperform GPUs on specific tasks, but are less flexible and depend on controlled environments and software stacks. section 8.1.2

Large training runs can distribute a model across thousands of accelerators. Parameters are split across devices, gradients are aggregated, and intermediate activations are stored or recomputed within memory limits. Throughput alone does not determine cost: networking bottlenecks, idle devices, and uneven workloads can leave expensive hardware underused. Sustained power, cooling, networking, and redundancy add further constraints. section 4.6.1

Figure 8.1 The large language model ecosystem. Models are built on a layered system that begins with specialized hardware and extends through cloud infrastructure and model providers. Interaction with this system occurs primarily through the two bottom layers, where users either consume models via managed services that abstract the underlying complexity or deploy open-weight models on self-managed infrastructure, taking on greater control and responsibility.

The ecosystem diagram places specialized hardware beneath cloud infrastructure and model providers, showing how users access models through managed services or self-managed infrastructure. section 8.1.2

How Machines that Think covers it

How deep it goes
Explains accelerator trade-offs and distributed training constraints without detailing chip architecture or implementation.
Examples
Contrasts GPUs, Google TPUs, AWS Trainium, and AWS Inferentia as hardware choices for machine learning.
What you can do after
Evaluate accelerator choices and large-scale deployment constraints in terms of flexibility, utilization, cost, and energy.
How it connects
Connects accelerators to distributed training, inference, parallel computing, cloud infrastructure, and energy consumption.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.