Why do stacked neural-network layers need more than matrix multiplication? An activation function adds nonlinearity, allowing the network to represent relationships that a single linear transformation cannot. section 2.4.2
A feed-forward layer computes y = f(x × w + b): weights multiply the incoming values, biases shift the result, and f applies the activation function. Without f, layered linear combinations collapse into one linear model, limiting classification to linearly separable problems such as excluding XOR. section 2.4 section 2.4.2
Common activation-function choices have different practical histories and properties:
- Sigmoid and hyperbolic tangent are smooth and differentiable at every point. section 2.4.2
- ReLU is not smooth at zero, yet practitioners found that networks often learn faster with it. section 2.4.2
- GELU is among the later designed activation functions. section 2.4.2
Using Swish in a JAX forward pass
from jax.nn import swish def predict(params, image): activations = image for w, b in params[:-1]: outputs = jnp.dot(w, activations) + b activations = swish(outputs) final_w, final_b = params[-1] logits = jnp.dot(final_w, activations) + final_b return logits Here, Swish transforms each hidden-layer result, while the final layer returns logits for the ten output classes. section 2.4.2 section 2.4
Listing 2.5 The forward pass of a neural network
import jax.numpy as jnp
from jax.nn import swish #1
def predict(params, image): #2
"""Function for per-example predictions."""
activations = image #3
for w, b in params[:-1]: #4
outputs = jnp.dot(w, activations) + b #5
activations = swish(outputs) #5
final_w, final_b = params[-1] #6
logits = jnp.dot(final_w, activations) + final_b #6
return logits
Inspect the complete forward-pass listing showing where the activation function sits between weighted layers. section 2.4.2
