liveBook · Manning

Pre-activation

As explained in Timeless Algorithms, The Foundational Ideas

A pre-activation value is the numerical result computed at a neural-network node before an activation function transforms it.

A pre-activation value is a node’s linear response before an activation function transforms it. For a hidden node, the value combines its input with its weight and bias. section 11.3.2

z1=w1x+b1

The node computes its pre-activation by multiplying the input by its weight and adding its bias. section 11.3.2

Two hidden nodes, one input

With input 2.0, the example’s upper node computes z₁ = 1.1 and its lower node computes z₂ = 0.1. These values are not probabilities or the signals passed onward; the sigmoid first transforms them into hidden-node outputs. section 11.3.2

Pre-activation values also provide the points through which the backward pass traces loss. Applying the chain rule links each value to its activated output and, in turn, to the loss. section 11.3.4

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines the linear calculation, works through numerical hidden-node values, and links them to gradient calculations without updating parameters.
Examples
Calculates two hidden-node values from input 2.0, then uses them in sigmoid activation and backward gradient calculations.
What it assumes
Uses inputs, weights, biases, activation functions, and loss in its calculations without establishing them from zero.
What you can do after
Understand how a node’s weighted input and bias produce a value before activation and how that value enters gradient calculations.
How it connects
Connects pre-activation to weights, biases, activation functions, sigmoid, and backpropagation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.