This chapter covers
- Understanding automatic differentiation
- Using automatic differentiation with PyTorch tensors
- Getting started with PyTorch SGD and Adam optimizers
- Applying PyTorch to linear regression with gradient descent
- Using data set batches for gradient descent
- PyTorch Dataset and DataLoader utility classes for batches
In chapter 5, you learned about the tensor, a core PyTorch data structure for n-dimensional arrays. The chapter illustrated the significant performance advantages of PyTorch tensors over native Python data structures for arrays and introduced PyTorch APIs for creating tensors as well as performing common operations on one or more tensors.
This chapter teaches another key feature of the PyTorch tensors: support for calculation of gradients using automatic differentiation (autodiff). Described as one of the major advances in scientific computing since 1970, autodiff is suprisingly simple and was invented by Seppo Linnainmaa, a master’s student at the University of Helsinki.1 The first part of this chapter introduces you to the fundamentals of autodiff by showing how you can implement the core algorithm for a scalar tensor using basic Python.