concept optimizer in category python

This is an excerpt from Manning's book Deep Learning with Python, Second Edition MEAP V04. Login to get full access to this book.

The fundamental trick in deep learning is to use this score as a feedback signal to adjust the value of the weights a little, in a direction that will lower the loss score for the current example (see figure 1.9). This adjustment is the job of the optimizer, which implements what’s called the Backpropagation algorithm: the central algorithm in deep learning. The next chapter explains in more detail how backpropagation works.

An optimizer — The mechanism through which the model will update itself based on the training data it sees, so as to improve its performance.

You’re nearing the end of this chapter, and you should now have a general understanding of what’s going on behind the scenes in a neural network. What was a magical black box at the start of the chapter has turned into a clearer picture, as illustrated in figure TODO: the model, composed of layers that are chained together, maps the input data to predictions. The loss function then compares these predictions to the targets, producing a loss value: a measure of how well the model’s predictions match what was expected. The optimizer uses this loss value to update the model’s weights.

Figure 2.24. Relationship between the network, layers, loss function, and optimizer

This is an excerpt from Manning's book Deep Learning with Python. Login to get full access to this book.

The fundamental trick in deep learning is to use this score as a feedback signal to adjust the value of the weights a little, in a direction that will lower the loss score for the current example (see figure 1.9). This adjustment is the job of the optimizer, which implements what’s called the Backpropagation algorithm: the central algorithm in deep learning. The next chapter explains in more detail how backpropagation works.

In this chapter, we’ll take a closer look at the core components of neural networks that we introduced in chapter 2: layers, networks, objective functions, and optimizers. We’ll give you a quick introduction to Keras, the Python deep-learning library that we’ll use throughout the book. You’ll set up a deep-learning workstation, with TensorFlow, Keras, and GPU support. We’ll dive into three introductory examples of how to use neural networks to address real problems:

The optimizer, which determines how learning proceeds

You can visualize their interaction as illustrated in figure 3.1: the network, composed of layers that are chained together, maps the input data to predictions. The loss function then compares these predictions to the targets, producing a loss value: a measure of how well the network’s predictions match what was expected. The optimizer uses this loss value to update the network’s weights.

Figure 3.1. Relationship between the network, layers, loss function, and optimizer

Let’s take a closer look at layers, networks, loss functions, and optimizers.

You’re passing your optimizer, loss function, and metrics as strings, which is possible because rmsprop, binary_crossentropy, and accuracy are packaged as part of Keras. Sometimes you may want to configure the parameters of your optimizer or pass a custom loss function or metric function. The former can be done by passing an optimizer class instance as the optimizer argument, as shown in listing 3.5; the latter can be done by passing function objects as the loss and/or metrics arguments, as shown in listing 3.6.

Listing 3.5. Configuring the optimizer

from keras import optimizers

model.compile(optimizer=optimizers.RMSprop(lr=0.001),
              loss='binary_crossentropy',
              metrics=['accuracy'])

copy

Optimizer— Determines how the network will be updated based on the loss function. It implements a specific variant of stochastic gradient descent (SGD).