A deep ConvNet can build useful feature hierarchies, but each additional block can make it harder for error information to reach earlier layers. When that signal becomes too weak or noisy, training can fail. A residual connection gives information a shortcut around a block: save its input, process the input, then add the saved value to the result. section 9.1 section 9.2
The basic move
In pseudocode: `residual = x; x = block(x); x = add([x, residual])`. The block can transform `x`, while the saved input provides a direct path past it. This is especially useful around blocks containing potentially destructive operations such as ReLU or dropout. section 9.2
Before adding the two paths, make their shapes match:
- If a block changes the number of filters, project the saved input with a 1 × 1 `Conv2D` layer without an activation. section 9.2
- If the block downsamples with max pooling, use strides in that projection to match the smaller spatial dimensions. `padding="same"` can keep the block’s convolutions from shrinking them unintentionally. section 9.2
Listing 9.3 The target block including a max pooling layer
inputs = keras.Input(shape=(32, 32, 3)) x = layers.Conv2D(32, 3, activation="relu")(inputs) residual = x #1 x = layers.Conv2D(64, 3, activation="relu", padding="same")(x) #2 x = layers.MaxPooling2D(2, padding="same")(x) #2 residual = layers.Conv2D(64, 1, strides=2)(residual) #3 x = layers.add([x, residual]) #4
The Keras listing shows both adjustments together: a convolution-and-pooling path is added to a 1 × 1, stride-2 projection of its saved input. section 9.2
A practical pattern is to repeat blocks of convolutions, optionally pool between them, and place a residual connection around each block. A mini Xception-like dogs-versus-cats ConvNet combines these shortcuts with batch normalization and separable convolutions. It reaches 90.8% test accuracy versus 83.9% for an earlier model, but that comparison does not isolate the effect of residual connections. section 9.2 section 9.5
Addition is not the only way to reconnect paths. A U-Net denoising autoencoder uses concatenative connections from downsampling blocks to corresponding upsampling blocks to help preserve image detail. Keep that use distinct from the additive shortcut used around an individual processing block. section 17.2.2 section 9.2
