A neuron first multiplies its inputs by weights and adds the results. An activation function transforms that weighted sum before it becomes the neuron’s output. Without this nonlinear step, the output is just a linear calculation. The choice of function depends on what the output needs to mean: a yes-or-no decision, a continuous likelihood, or a set of labels. section 5.1.5 section 8.2.4
From word counts to a decision
Count the words in “green egg egg ham ham ham spam spam spam spam” to get the input vector [1, 2, 3, 4]. With the illustrated weights and bias, the weighted sum is −0.76. A step activation with threshold zero turns it into 0; a value above zero would become 1. Use a logistic activation instead when the neuron should produce a continuous likelihood rather than a binary result. A single logistic neuron is mathematically equivalent to a logistic-regression model, although the training methods differ. section 5.1.5 section 5.2
>>> from collections import Counter >>> np.random.seed(451) >>> tokens = "green egg egg ham ham ham spam spam spam spam".split() >>> bow = Counter(tokens) >>> x = pd.Series(bow) >>> x green 1 egg 2 ham 3 spam 4 >>> x1, x2, x3, x4 = x >>> x1, x2, x3, x4 (1, 2, 3, 4) >>> w0 = np.round(.1 * np.random.randn(), 2) >>> w0 0.07 >>> w1, w2, w3, w4 = (.1 * np.random.randn(len(x))).round(2) >>> w1, w2, w3, w4 (0.12, -0.16, 0.03, -0.18) >>> x = np.array([1, x1, x2, x3, x4]) #1 >>> w = np.array([w0, w1, w2, w3, w4]) #2 >>> y = np.sum(w * x) #3 >>> y -0.76
Inspect the Python calculation before activation: word counts become an input array, which combines with weights to produce −0.76. section 5.1.5
The same choice appears in larger NLP systems:
- In a convolutional neural network, a kernel scores successive windows of text. Max pooling selects the strongest match, and an activation function converts it to a binary value or likelihood. section 7.2.1
- For mutually exclusive classes, softmax produces a probability distribution across labels. For multi-label tagging, an element-wise sigmoid gives each label its own value between zero and one. Switching the activation is therefore essential when changing a multiclass classifier into a multi-label tagger. section 8.2.4
