Sigmoid

As explained in Math and Architectures of Deep Learning

An S-shaped nonlinear activation function whose steepness and horizontal position can be adjusted with parameters.

The sigmoid is an S-shaped nonlinear activation function, commonly written as σ. Applied to a weighted sum, it adds nonlinearity so a model can represent classification boundaries that a linear weighted combination cannot. section 1.6

A parameterized sigmoid changes shape through two parameters: w adjusts the steepness of the central portion, while b shifts the curve horizontally. Increasing w makes the curve increasingly step-like, while preserving differentiability. section 8.1.1

Its geometry explains both its usefulness and its training behavior:

• The curve climbs most rapidly through its middle region, where its derivative is largest. section 8.1.1
• The curve saturates toward both ends, becoming nearly flat and producing derivatives close to zero. section 8.1.1
• Unlike the Heaviside step function, the sigmoid is differentiable, making gradient-based neural-network training possible. section 8.1 section 8.1.1

Nonlinear classification

A weighted sum alone produces a linear separator. Passing that weighted sum through a sigmoid bends the model’s response and supports more complex classification tasks, forming a basic neural-network building block. section 1.6

Figure 8.2 Sigmoid curves corresponding to various parameter values in equation 8.2

Compare how changing w and b alters the sigmoid’s steepness and horizontal position. section 8.1.1

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines the curve, parameter effects, differentiability, and derivative behavior, but does not provide implementation code.

Examples

Uses weighted-sum classification, parameterized curves, derivative-overlaid graphs, and transformed manifolds as examples.

What it assumes

Builds from weighted sums, linear separators, the Heaviside step function, derivatives, and gradient-based training.

What you can do after

Evaluate how sigmoid steepness, position, saturation, and differentiability affect neural-network classification and training.

How it connects

Connects nonlinear models, neural networks, the Heaviside step function, gradients, and homeomorphic transformations.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA function that maps a real-valued input to a value between zero and one, commonly interpreted as a binary class probability.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA nonlinear activation function that maps real-valued inputs to a bounded interval.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsA sigmoid function is a function with an S-shaped curve that transitions smoothly between lower and upper limiting values.
• 

  liveBookmanning.com
  Grokking StatisticsAn S-curve is a sigmoid-shaped relationship in which growth is initially rapid, then slows toward a plateau.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA sigmoid function is a bounded S-shaped mathematical function that maps real-valued inputs to values between two limits, commonly zero and one.
• 

  liveBookmanning.com
  Data Without LabelsA sigmoid function is a differentiable S-shaped function that maps real-valued inputs to bounded output values and is commonly used to produce binary-classification scores.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA logistic function is an S-shaped mathematical function that maps an input to a bounded continuous value and provides a soft threshold.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionA sigmoid activation maps a real-valued score to a value between zero and one, commonly interpreted as a probability for a binary outcome.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA sigmoid is a smooth mathematical function that maps real-valued inputs to values between zero and one.
• 

  liveBookmanning.com
  How Large Language Models WorkAn S-curve is a growth pattern in which progress accelerates, reaches a transition region, and then approaches a plateau.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchAn activation function that maps outputs into the range 0 to 1, used here to interpret the discriminator output as a probability.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA sigmoid activation function maps arbitrary real-valued inputs to values in the interval from zero to one.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA sigmoid function is a continuous S-shaped function that maps real-valued inputs to values between zero and one.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA statistical classification method that models class probabilities with a linear predictor passed through a logistic function.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackThe sigmoid function maps a real-valued input to a number between zero and one using the function 1/(1+e^-x).

Compare how 16 books explain sigmoid →

Related concepts

• activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
• logistic regressionused by · Logistic regression is a statistical classification model that estimates categorical outcome probabilities by applying a logistic or softmax function to a linear combination of input features.
• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• probability theoryproduces · Probability theory is the branch of mathematics that formalizes uncertainty and analyzes random events and outcomes using probability measures.
• cross entropy lossused by · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.