A computation graph represents a calculation as a directed acyclic graph of operations. In a neural network, those operations act on tensors. Making the calculation an explicit data structure lets another program work with it—for example, to generate derivatives automatically. section 2.4.4
From a prediction to a loss
Consider a model that applies matrix multiplication, bias addition, ReLU, another matrix multiplication and bias addition, then softmax and a loss function. The graph records the operations connecting the inputs and weights to the loss. A forward pass computes the result; a backward pass follows the operations that contributed to it to compute gradients. section 2.4.4
The key rule is the chain rule. If x becomes x1 through g, then x1 becomes y through f, the gradient of y with respect to x is the product of the gradient of y with respect to x1 and the gradient of x1 with respect to x. Backpropagation applies this rule across the graph to find gradients of neural-network weights. section 2.4.4
loss_value = loss( y_true, softmax(matmul(relu(matmul(inputs, W1) + b1), W2) + b2), )
The nested loss expression shows the tensor operations that can be represented as a computation graph. section 2.4.4
In PyTorch, the graph is usually invisible: running a computation creates a one-time record of the operations. Call backward() on the result to propagate gradients into the .grad attributes of tensors tracking them. Repeated calls add to existing gradients, so set .grad to None when you need to reset them. section 3.4.1
Listing 3.26 Computing a gradient with .backward()
>>> input_var = torch.tensor(3.0, requires_grad=True) #1 >>> result = torch.square(input_var) >>> result.backward() #2 >>> gradient = input_var.grad #2 >>> gradient tensor(6.)
A scalar square makes the workflow concrete: mark the input for gradient tracking, compute the result, call backward(), and read input_var.grad. section 3.4.1
Do not confuse a graph of operations with a model’s graph of layers. The latter describes network topology: how layers connect and which sequence of tensor operations the model can use. In Keras, a simple Sequential model is a stack of layers; other topologies can include branches or residual connections. section 2.4.4 section 3.6.3
