How deeply does Deep Learning with Python, Third Edition cover tf.GradientTape?

Works through variable gradients, explicitly watched constants, nested tapes for second-order gradients, and complete training and gradient-ascent steps.

Tf.GradientTape

As explained in Deep Learning with Python, Third Edition

TensorFlow’s context object for recording tensor operations so gradients can later be computed from the recorded graph.

To update a model, you need to know how its loss changes with its weights. `tf.GradientTape` records TensorFlow operations performed inside its scope so you can compute those gradients afterward. Put the forward calculation inside `with tf.GradientTape() as tape:`, then call `tape.gradient(output, inputs)` outside it. section 3.3.1 section 7.4.2

Start with a square

Create `input_var = tf.Variable(3.0)`. Inside the tape scope, calculate `result = tf.square(input_var)`. Afterward, `tape.gradient(result, input_var)` gives the gradient of the result with respect to the variable. The important boundary is the tape scope: calculate the result while recording, then ask for its gradient. section 3.3.1

Two useful rules when moving beyond that first example:

Variables work directly as gradient inputs. For a constant tensor, call `tape.watch(input_const)` inside the scope before calculating the result. section 3.3.1
For a second-order gradient, nest two tapes: use the inner tape to differentiate once and the outer tape to differentiate that result. section 3.3.1

import tensorflow as tf

model = get_mnist_model()
loss_fn = keras.losses.SparseCategoricalCrossentropy()
optimizer = keras.optimizers.Adam()

def train_step(inputs, targets):
    with tf.GradientTape() as tape:                               #1
        predictions = model(inputs, training=True)                #2
        loss = loss_fn(targets, predictions)                       #2
    gradients = tape.gradient(loss, model.trainable_weights)      #3
    optimizer.apply(gradients, model.trainable_weights)           #4
    return loss

The complete TensorFlow training step places model predictions and loss calculation inside the tape, then obtains gradients with respect to `model.trainable_weights` and passes them to the optimizer. Use this pattern when writing a custom Keras training step. section 7.4.2

The same pattern can optimize an input instead of model weights. To increase a convolutional filter’s activation, watch the image, compute the activation-based loss inside the tape, and take its gradient with respect to the image. A `@tf.function` decorator can speed up the step. section 10.2.1

How Deep Learning with Python, Third Edition covers it

Cover of Deep Learning with Python, Third Edition

liveBook manning.com

Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Deep Learning with R, Third Editiontf.GradientTape is a TensorFlow context-based API object that records tensor operations for subsequent gradient computation.

Compare how 2 books explain tf.GradientTape →

Related concepts

gradientproduces · A gradient is a vector or tensor of partial derivatives that describes how a function changes with respect to its input variables or parameters.
TensorFlowused by · TensorFlow is an open-source software platform for numerical computation and machine learning, with tools for building, training, and deploying models.
computation graphproduces · A computation graph is a directed graph whose nodes represent computational operations or values and whose edges represent data dependencies between them.
forward passuses · A forward pass is a calculation that propagates inputs through an ordered computational or dependency structure to produce outputs or derived values.
training stepused by · A training step is an iteration in which a model is updated during training using data and an optimization objective.
automatic differentiationimplements · Automatic differentiation is a computational technique that evaluates derivatives of a program by systematically applying the chain rule to its elementary operations.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.