ReLU (Rectified Linear Unit)

ReLU, or Rectified Linear Unit, is a fundamental activation function used extensively in deep learning models. It is known for its simplicity and effectiveness in introducing non-linearity into neural networks, which is crucial for learning complex patterns. The ReLU function is defined as:

[ f(x) = \max(0, x) ]

This means that for any input ( x ), the output is ( x ) if ( x ) is positive, and zero otherwise. This characteristic allows ReLU to mitigate the vanishing gradient problem, which can occur with other activation functions like sigmoid or tanh, making it a popular choice in various neural network architectures.

Characteristics of ReLU

ReLU is a piecewise linear function that outputs the input directly if it is positive and zero for any negative input. This simplicity contributes to its computational efficiency, as it requires only a simple thresholding at zero. The function’s ability to maintain non-linearity while being computationally efficient makes it a preferred choice in the design of deep learning models.

Visualization

The behavior of the ReLU function can be visualized in comparison to other activation functions, such as GELU (Gaussian Error Linear Unit). The following figure illustrates the output of both the GELU and ReLU functions:

Figure 4.8 The output of the GELU and ReLU plots using matplotlib. The x-axis shows the function inputs and the y-axis shows the function outputs.

In this plot, the x-axis represents the function inputs, while the y-axis represents the function outputs. The ReLU function is depicted as a linear increase for positive inputs and a flat line at zero for negative inputs, highlighting its straightforward and efficient nature.

ReLU in Neural Network Architectures

Basic Usage

ReLU is often used in convolutional neural networks (CNNs) and fully connected networks. Below is an example of a simple neural network using ReLU as the activation function:

class NetDepth(nn.Module):
    def __init__(self, n_chans1=32):
        super().__init__()
        self.n_chans1 = n_chans1
        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(n_chans1, n_chans1 // 2, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(n_chans1 // 2, n_chans1 // 2, kernel_size=3, padding=1)
        self.fc1 = nn.Linear(4 * 4 * n_chans1 // 2, 32)
        self.fc2 = nn.Linear(32, 2)

    def forward(self, x):
        out = F.max_pool2d(torch.relu(self.conv1(x)), 2)
        out = F.max_pool2d(torch.relu(self.conv2(out)), 2)
        out = F.max_pool2d(torch.relu(self.conv3(out)), 2)
        out = out.view(-1, 4 * 4 * self.n_chans1 // 2)
        out = torch.relu(self.fc1(out))
        out = self.fc2(out)
        return out

In this example, ReLU is applied after each convolutional layer and the first fully connected layer.

ReLU with Skip Connections

ReLU can also be used in more advanced architectures like ResNets, which incorporate skip connections. Skip connections help in alleviating the vanishing gradient problem by allowing gradients to flow through the network more easily.

class NetRes(nn.Module):
    def __init__(self, n_chans1=32):
        super().__init__()
        self.n_chans1 = n_chans1
        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(n_chans1, n_chans1 // 2, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(n_chans1 // 2, n_chans1 // 2, kernel_size=3, padding=1)
        self.fc1 = nn.Linear(4 * 4 * n_chans1 // 2, 32)
        self.fc2 = nn.Linear(32, 2)

    def forward(self, x):
        out = F.max_pool2d(torch.relu(self.conv1(x)), 2)
        out = F.max_pool2d(torch.relu(self.conv2(out)), 2)
        out1 = out
        out = F.max_pool2d(torch.relu(self.conv3(out)) + out1, 2)
        out = out.view(-1, 4 * 4 * self.n_chans1 // 2)
        out = torch.relu(self.fc1(out))
        out = self.fc2(out)
        return out

In this model, a skip connection is added by summing the output of the first layer with the input to the third layer.

Deep Residual Networks with ReLU

For deeper networks, ReLU is used in conjunction with batch normalization and custom initializations to stabilize training:

class ResBlock(nn.Module):
    def __init__(self, n_chans):
        super(ResBlock, self).__init__()
        self.conv = nn.Conv2d(n_chans, n_chans, kernel_size=3, padding=1, bias=False)
        self.batch_norm = nn.BatchNorm2d(num_features=n_chans)
        torch.nn.init.kaiming_normal_(self.conv.weight, nonlinearity='relu')
        torch.nn.init.constant_(self.batch_norm.weight, 0.5)
        torch.nn.init.zeros_(self.batch_norm.bias)

    def forward(self, x):
        out = self.conv(x)
        out = self.batch_norm(out)
        out = torch.relu(out)
        return out + x

In this ResBlock, ReLU is applied after batch normalization, and the output is added to the input to form a skip connection.

class NetResDeep(nn.Module):
    def __init__(self, n_chans1=32, n_blocks=100):
        super().__init__()
        self.n_chans1 = n_chans1
        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
        self.resblocks = nn.Sequential(*(n_blocks * [ResBlock(n_chans=n_chans1)]))
        self.fc1 = nn.Linear(8 * 8 * n_chans1, 32)
        self.fc2 = nn.Linear(32, 2)

    def forward(self, x):
        out = F.max_pool2d(torch.relu(self.conv1(x)), 2)
        out = self.resblocks(out)
        out = F.max_pool2d(out, 2)
        out = out.view(-1, 8 * 8 * self.n_chans1)
        out = torch.relu(self.fc1(out))
        out = self.fc2(out)
        return out

In this deeper network, multiple ResBlock instances are used in sequence, demonstrating the scalability of ReLU in deep architectures.

For more detailed information, you can refer to the original discussions in Build a Large Language Model (From Scratch) and Deep Learning with PyTorch, Second Edition.