liveBook · Manning

Neural network

As explained in Timeless Algorithms, The Foundational Ideas

A neural network is a parameterized computational model composed of interconnected layers that transform inputs through learned functions.

A neural network transforms inputs such as images, sensor readings, or word embeddings through layers of interconnected computational units to produce a prediction. Its adjustable weights and biases are learned from data. Successive layers can learn internal representations, reducing reliance on features specified in advance. section 11.2 section 13.4.2

In a feedforward network, a node combines its inputs as z = ∑ᵢwᵢxᵢ + b and produces an output a = f(z). Here xᵢ are incoming signals, wᵢ are weights, b is a bias, and f is an activation function. Nonlinear activations allow the network to represent relationships that successive linear transformations alone cannot capture. Depth determines how many transformations are composed; width determines how many patterns can be represented at a given stage. section 11.2 section 11.8

Figure 11.1 A simplified feedforward neural network containing an input layer, two hidden layers, and an output layer. Information moves forward through weighted connections to produce a prediction. During training, error information moves backward so that the weights and biases can be adjusted. Modern deep networks typically contain many hidden layers and many nodes within each layer, producing millions or even billions of adjustable parameters. Despite that scale, they follow the same basic forward-and-backward process shown here.

The diagram traces a forward prediction through a feedforward network and the backward flow of error information during training. section 11.2

Training repeats a forward pass, loss calculation, backward pass, and parameter update. The loss measures the discrepancy between prediction and target. Backpropagation applies the chain rule in reverse to calculate gradients for the weights and biases; an optimizer uses those gradients to adjust the parameters. A single loss can guide changes throughout the network, including its internal representations. After training, prediction requires only the forward pass. section 11.2 section 11.8

The output and loss depend on the task. Regression produces a continuous value and often uses squared-error loss. Classification can produce probabilities and commonly uses cross-entropy. A binary probability can be compared with a threshold to make a class decision; the probability, the resulting class, and any subsequent action are distinct. section 11.2 section 11.8

Many common losses connect neural-network training to maximum likelihood. Classification cross-entropy is the negative log-likelihood of a categorical probability model; squared error corresponds to negative log-likelihood under normally distributed errors. The loss therefore embodies an assumption about the data, even when optimization is performed by repeated gradient updates rather than a closed-form solution. section 3.5 section 5.4.1

More depth and width increase representational capacity but also increase the parameters to estimate. Training depends on data, computing resources, initialization, and optimization choices. Neural-network loss surfaces are generally non-convex, so gradient descent has no general guarantee of finding a global minimum. Lower training loss also does not guarantee better performance on unseen observations. Predictions can reflect biased data or express confidence greater than their accuracy warrants. section 11.5.3 section 11.6.1 section 11.6

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Traces the forward pass, loss, backpropagation, and parameter updates, then examines optimization and generalization limits.
Examples
Uses a feedforward-network diagram and contrasts regression outputs with binary and multiclass predictions.
What you can do after
Evaluate and understand neural-network predictions, training objectives, and limitations rather than treat the model as a black box.
How it connects
Connects representation learning with maximum likelihood, cross-entropy, gradient descent, and generalization.

Also explained in

Compare how 11 books explain neural network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.