ReLU, short for rectified linear unit, outputs zero for negative inputs and returns the input itself for positive inputs. It is widely used as a general activation function and appears between convolutional layers in neural-network blocks. section 6.1.4 section 13.4.1
ReLU inside a convolutional block
A PyTorch block can apply ReLU after each convolution, then reduce the result with max pooling: `block_out = self.conv1(input_batch); block_out = self.relu1(block_out); block_out = self.conv2(block_out); block_out = self.relu2(block_out); return self.maxpool(block_out)`. The layers are created with `nn.ReLU(inplace=True)`. section 13.4.1
Listing 13.5 Defining the LunaBlock model (model.py)
class LunaBlock(nn.Module):
def __init__(self, in_channels, conv_channels):
super().__init__()
self.conv1 = nn.Conv3d(
in_channels, conv_channels, kernel_size=3, padding=1, bias=True,
)
self.relu1 = nn.ReLU(inplace=True)
self.conv2 = nn.Conv3d(
conv_channels, conv_channels, kernel_size=3, padding=1, bias=True,
)
self.relu2 = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool3d(2, 2)
def forward(self, input_batch):
block_out = self.conv1(input_batch)
block_out = self.relu1(block_out)
block_out = self.conv2(block_out)
block_out = self.relu2(block_out)
return self.maxpool(block_out)
Inspect the complete LunaBlock listing to see ReLU layers positioned after two 3 × 3 convolutions and before max pooling. section 13.4.1
