liveBook · Manning

Sigmoid

As explained in Grokking Machine Learning, Second Edition

The sigmoid function maps a real-valued input smoothly to a value between zero and one, commonly interpreted as a binary-class probability.

A sigmoid turns a score into a smooth number between 0 and 1. Positive scores move toward 1, negative scores move toward 0, and a score of 0 becomes 0.5. This makes a classifier express degrees of confidence instead of only yes-or-no decisions. section 6.2 section 6.0

Figure 6.2 The sigmoid function sends the entire number line to the interval (0,1).

The curve squeezes the whole number line into the interval between 0 and 1. section 6.2

For a logistic classifier, use the sigmoid in three small steps:

  • Compute a score by multiplying each feature by its weight, adding the results, and adding a bias. section 6.11
  • Apply the sigmoid: σ(x) = exp(x) / (1 + exp(x)). The result is the prediction. section 6.11
  • Interpret predictions above or equal to 0.5 as one class and lower predictions as the other when using that threshold. section 6.3

Scoring an alien-language sentence

With weights 1 for “aack,” 2 for “beep,” and bias −4, a sentence containing three aacks and two beeps gets score 3 + 2×2 − 4 = 3. Its sigmoid prediction is 0.953: close to the happy end, even though the label can still reveal a mistake. section 6.3

Sigmoid replaces the discontinuous step function in a continuous perceptron. Its continuous, differentiable shape supplies richer predictions and supports the calculus used while training. In neural networks, it can serve as an activation function, especially at an output layer; its flat derivatives can also contribute to vanishing gradients. section 6.2 section 10.25

def sigmoid(x):
    return np.exp(x)/(1+np.exp(x))

This compact NumPy implementation converts one numeric score into a sigmoid prediction. section 6.11

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines the mapping, contrasts it with the step function, and implements it inside logistic regression without deriving calculus details.
Examples
Uses happy/sad alien-language sentences, numeric sigmoid values, a weighted perceptron, and plotted classifier boundaries.
What it assumes
Assumes familiarity with perceptron classifiers and their iterative training procedure.
What you can do after
Build it from scratch as a Python sigmoid, then apply it to weighted scores in logistic classification.
How it connects
Connects logistic regression, perceptrons, step functions, activation functions, log loss, neural networks, and vanishing gradients.

Also explained in

Compare how 23 books explain sigmoid →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.