An artificial neuron turns numerical inputs into one output. Each input has a weight that controls its influence; the neuron adds the weighted inputs and applies an activation function. It is a small computational unit, not a miniature human brain. section 5.1.3 section 5.1.5
For text, the inputs must first become numbers. A bag-of-words vector might record word counts; a TF-IDF vector can represent thousands of text features. One neuron can then learn which features matter to its prediction instead of treating every word count alike. section 5.1.5 section 5.2.5
A neuron in a few lines of Python
Given input values x and matching weights w, compute z = sum(xi * wi for xi, wi in zip(x, w)). Include a constant input of 1 to give the neuron an adjustable bias. Returning z > 0 makes a threshold decision; applying a logistic function instead produces a value suitable for a binary-classification probability. A single neuron with logistic activation is mathematically equivalent to logistic regression, though its training can differ. section 5.1.5 section 5.2
>>> from collections import Counter >>> np.random.seed(451) >>> tokens = "green egg egg ham ham ham spam spam spam spam".split() >>> bow = Counter(tokens) >>> x = pd.Series(bow) >>> x green 1 egg 2 ham 3 spam 4 >>> x1, x2, x3, x4 = x >>> x1, x2, x3, x4 (1, 2, 3, 4) >>> w0 = np.round(.1 * np.random.randn(), 2) >>> w0 0.07 >>> w1, w2, w3, w4 = (.1 * np.random.randn(len(x))).round(2) >>> w1, w2, w3, w4 (0.12, -0.16, 0.03, -0.18) >>> x = np.array([1, x1, x2, x3, x4]) #1 >>> w = np.array([w0, w1, w2, w3, w4]) #2 >>> y = np.sum(w * x) #3 >>> y -0.76
Inspect the word-count vector, bias input, weights, and resulting weighted sum before an activation function is applied. section 5.1.5
A practical application uses character n-grams from first names to predict a label recorded on US birth certificates. The label is sex assigned at birth, not a person’s gender, and a name does not have one universally correct binary label. Treat a prediction as uncertain rather than as a fact about someone. section 5.2.2
>>> import torch >>> class LogisticRegressionNN(torch.nn.Module): ... def __init__(self, num_features, num_outputs=1): ... super().__init__() ... self.linear = torch.nn.Linear(num_features, num_outputs) ... def forward(self, X): ... return torch.sigmoid(self.linear(X)) >>> model = LogisticRegressionNN(num_features=vecs.shape[1], num_outputs=1) >>> model LogisticRegressionNN( (linear): Linear(in_features=3663, out_features=1, bias=True) )
See how a PyTorch module combines a linear layer with sigmoid activation to make a single-output logistic neuron. section 5.2.5
To train that PyTorch neuron, compare its outputs with labels using binary cross-entropy, then backpropagate the loss and update its parameters with an optimizer. Check performance on held-out data: even a useful single-neuron classifier can fail on individual cases, and NLP systems can be brittle outside their test conditions. section 5.2.5 section 5.1.1
