An accelerator is a processor designed to execute particular workloads efficiently. In large language models, accelerators handle the repeated, large-scale matrix operations involved in training and the computation needed to produce responses. section 4.6.1 section 8.1.2
Training repeatedly updates model parameters through forward and backward passes, gradient computations, and optimization. Inference uses frozen parameters and typically requires only a forward pass. It is less expensive than training, though it can still be intensive at scale. section 4.6.1
Accelerator designs make different trade-offs:
- GPUs perform many operations in parallel. Their flexibility and mature software ecosystem have made them standard for training and running modern AI models, though highly specialized workloads can expose efficiency limits. section 8.1.2
- TPUs and custom accelerators such as AWS Trainium and AWS Inferentia optimize computation and data flow for machine learning. They can outperform GPUs on specific tasks, but are less flexible and depend on controlled environments and software stacks. section 8.1.2
Large training runs can distribute a model across thousands of accelerators. Parameters are split across devices, gradients are aggregated, and intermediate activations are stored or recomputed within memory limits. Throughput alone does not determine cost: networking bottlenecks, idle devices, and uneven workloads can leave expensive hardware underused. Sustained power, cooling, networking, and redundancy add further constraints. section 4.6.1
Figure 8.1 The large language model ecosystem. Models are built on a layered system that begins with specialized hardware and extends through cloud infrastructure and model providers. Interaction with this system occurs primarily through the two bottom layers, where users either consume models via managed services that abstract the underlying complexity or deploy open-weight models on self-managed infrastructure, taking on greater control and responsibility.

The ecosystem diagram places specialized hardware beneath cloud infrastructure and model providers, showing how users access models through managed services or self-managed infrastructure. section 8.1.2
