ReLU (Rectified Linear Unit) is an element-wise activation that returns each input when it is positive and replaces negative values with zero. In a neural-network layer, it follows the matrix multiplication and bias addition that produce an intermediate representation. section 2.3 section 4.1.3
ReLU inside a Dense layer
A Dense layer can be read as output = relu(dot(input, W) + b). The dot product projects the input into a representation space, the bias shifts it, and ReLU zeroes the negative entries. A practical Keras stack uses ReLU in intermediate layers and sigmoid only for the final binary-classification output. section 4.1.3 section 2.3
Use these rules when implementing or recognizing ReLU:
- Treat it as an element-wise operation: each tensor entry is processed independently. section 2.3.1
- Use optimized array operations such as NumPy's maximum rather than nested Python loops for practical performance. section 2.3.1
- Expect the same operation to map efficiently onto parallel CPU or GPU implementations. section 2.3.1
- Recognize stacks of Dense layers with ReLU activations as a broadly useful baseline, including for sentiment classification. section 4.1.7
def naive_relu(x): assert len(x.shape) == 2 #1 x = x.copy() #2 for i in range(x.shape[0]): for j in range(x.shape[1]): x[i, j] = max(x[i, j], 0) return x
Inspect the direct two-dimensional Python implementation to see ReLU applied independently to every tensor entry. section 2.3.1
