Chapter 13. Introducing automatic optimization: let’s build a deep learning framework

“Whether we are based on carbon or on silicon makes no fundamental difference; we should each be treated with appropriate respect.”

Arthur C. Clarke, 2010: Odyssey Two (1982)

What is a deep learning framework?

Good tools reduce errors, speed development, and increase runtime performance

If you’ve been reading about deep learning for long, you’ve probably come across one of the major frameworks such as PyTorch, TensorFlow, Theano (recently deprecated), Keras, Lasagne, or DyNet. Framework development has been extremely rapid over the past few years, and, despite all frameworks being free, open source software, there’s a light spirit of competition and comradery around each framework.

Introduction to tensors

Introduction to automatic gradient computation (autograd)

A quick checkpoint

Tensors that are used multiple times

Upgrading autograd to support multiuse tensors

How does addition backpropagation work?

Adding support for negation

Adding support for additional functions

Using autograd to train a neural network

Adding automatic optimization

Adding support for layer types

Layers that contain layers

Loss-function layers

How to learn a framework

Nonlinearity layers

The embedding layer

Adding indexing to autograd

The embedding layer (revisited)

The cross-entropy layer

The recurrent neural network layer

Summary