An activation function applies a nonlinear transformation after a linear operation. Without these nonlinearities, stacking linear layers cannot build more complex functions. section 2.2.3
Common choices differ in output range and gradient behavior:
- Sigmoid, σ(x), maps inputs to [0, 1]. section 2.2.3
- Tanh maps inputs to [−1, 1] and remains approximately linear around zero. section 2.2.3
- ReLU returns x for positive inputs and zero for negative inputs; its simple derivative often supports faster convergence than tanh or sigmoid. section 6.1.2
- Leaky ReLU scales negative inputs by α, preserving a small gradient where ReLU has none; α values from 0.01 to 0.3 are discussed. section 6.1.2
Insert an activation in PyTorch
Place an activation between hidden linear layers, while typically ending with a linear output layer: nn.Sequential(nn.Linear(1, 10), nn.Tanh(), nn.Linear(10, 1)). Replace nn.Tanh() with a ReLU or LeakyReLU when testing gradient and convergence tradeoffs. section 2.2.3 section 6.1.2
activation_input = np.linspace(-2, 2, num=200)
tanh_activation = np.tanh(activation_input)
sigmoid_activation = np.exp(activation_input)/(np.exp(activation_input)+1)
sns.lineplot(x=activation_input, y=activation_input,
color=’black’, label="linear")
sns.lineplot(x=activation_input, y=tanh_activation,
color=’red’, label="tanh(x)")
ax = sns.lineplot(x=activation_input, y=sigmoid_activation,
color=’blue’, label="$\sigma(x)$)")
ax.set_xlabel(’Input value x’)
ax.set_ylabel(’Activation’)
[13]: Text(0, 0.5, 'Activation')
Plot linear, tanh, and sigmoid curves across input values to compare their ranges and nonlinear shapes. section 2.2.3
model = nn.Sequential( nn.Linear(1, 10), #1
nn.Tanh(), #2
nn.Linear(10, 1), #3
)
train_simple_network(model, loss_func, training_loader, epochs=200)
with torch.no_grad():
Y_pred = model(torch.tensor(X.reshape(-1,1),
dtype=torch.float32)).cpu().numpy()
sns.scatterplot(x=X, y=y, color=’blue’, label=’Data’) #4
sns.lineplot(x=X, y=Y_pred.ravel(), color=’red’, label=’Model’) #5
[15]: <AxesSubplot:>
#1 Hidden layer
#2 Activation
#3 Output layer
#4 The data
#5 What our model learned
Inspect a complete PyTorch model that places nn.Tanh between a hidden layer and the output layer, then plots predictions. section 2.2.3
