Sigmoid is a nonlinear activation that lets neural networks represent functions beyond stacked linear layers. It maps every real-valued input into the bounded interval [0,1]. section 2.2.3
The notation σ(x) commonly denotes sigmoid. A direct implementation is σ(x) = exp(x) / (exp(x) + 1); plotting it alongside a linear function makes its saturation and bounded output visible. section 2.2.3
Use sigmoid when its bounded output has a clear modeling role:
- Place it after a linear operation to introduce nonlinearity, while recognizing that sigmoid and tanh can produce vanishing gradients. section 2.2.3 section 6.1
- Use it as an LSTM gate: multiplying a value by σ produces a controllable flow from zero, through partial transmission, to the full value. section 6.6.2
- End a GAN generator with nn.Sigmoid when generated outputs must match data constrained to [0,1], such as MNIST images. section 9.1.3
PyTorch placement
A hidden sigmoid can occupy the same position as the demonstrated tanh layer: nn.Sequential(nn.Linear(1, 10), nn.Sigmoid(), nn.Linear(10, 1)). The final linear layer remains the output layer. section 2.2.3
Gating intuition
If z = α · σ(·), a sigmoid value of 0 closes the gate, 1 passes α unchanged, and intermediate values regulate how much information flows. section 6.6.2
