A bias is a learned additive parameter that shifts a layer’s output. In a simple linear model, the bias is the output value when every input is zero, while the weight controls how strongly the input influences the result. section 5.2.4
A neuron first scales its input, adds the bias, and then applies an activation function: o = f(w × x + b). For a layer, the bias can be a scalar or a vector matching the output dimensions. section 6.1
Bias appears in several closely related settings:
- Linear models learn a weight and a bias together to fit measured outputs. section 5.2.4
- Neural-network neurons add bias before applying a nonlinear activation. section 6.1
- A two-dimensional convolution has one bias value for each output channel, added to that channel’s output. section 8.2
A linear model
The model t_c = w * t_u + b converts an unknown temperature measurement into Celsius. Training changes w and b to reduce the loss; after normalization and extended training, the learned parameters are approximately [5.3671, -17.3012]. section 5.2.4 section 5.4.4
# In[8]: conv.weight.shape, conv.bias.shape # Out[8]: (torch.Size([16, 3, 3, 3]), torch.Size([16]))
Inspecting a convolution shows that its weight tensor has shape [16, 3, 3, 3], while its bias has shape [16]: one learned additive value per output channel. section 8.2
