liveBook · Manning

Chapter 3: Artificial neural networks

What you will learn in chapter 3 of Machines that Think

This chapter introduction frames neural networks as multi-stage systems that build structured representations from complex data and can be architected for different kinds of patterns.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This chapter introduction frames neural networks as multi-stage systems that build structured representations from complex data and can be architected for different kinds of patterns.

Key terms: neural networks, high-dimensional data, internal representations, successive stages of computation, architecture, spatial structure, sequential dependencies, distant parts of an input, coherent, adaptive

3.1 What is a neural network?

This section introduces neural networks as layered systems of interconnected computations that transform numerical inputs into increasingly complex representations.

Key terms: neural network, inputs, outputs, interconnected computations, numerical inputs, relative influence, output, transformation, successive stages, complex representations

3.1.1 Neurons as a web of mathematics

This section defines an artificial neuron as a simplified mathematical unit that weights numerical inputs, adds a bias, applies a nonlinear activation function, and produces an output for a larger network.

Key terms: artificial neuron, inputs, output, vector, features, weights, bias, weighted combination, activation function, nonlinear

3.1.2 Layers that shape information

This section explains how neural-network layers transform inputs into outputs and why greater depth enables progressively more abstract pattern learning.

Key terms: input layer, hidden layers, output layer, signals, layer, learned representations, intermediate features, raw data, deep learning, shallow network

3.1.3 Weighing what is important

This section explains how adjustable weights store a neural network’s distributed learned influence, enabling pattern recognition while making internal behavior difficult to interpret.

Key terms: weights, inputs, neuron, output, signals, training, examples, useful outcomes, patterns, distributed map of influence

3.1.4 Shaping the neuron’s response

This section explains how activation functions add nonlinearity to neural networks, enabling deep layers to represent complex patterns rather than collapsing into one linear calculation.

Key terms: activation functions, nonlinearities, weighted sums, raw output, expressive transformations, linear transformation, layers, specialize, sigmoid, tanh

3.1.5 Tracing the journey

This section traces how a neural network transforms numerical inputs through weighted, activated layers into an output estimate via a forward pass.

Key terms: forward pass, numerical inputs, learned weights, activation functions, output, early layers, later layers, simple patterns, abstract cues, structured response

3.1.6 Becoming the dominant paradigm

This section explains why neural networks became modern AI's dominant paradigm: they scale with data and compute while learning task-specific representations end-to-end.

Key terms: data, computing power, simple components, deep architectures, modern AI, predefined way of describing the data, transform inputs, prediction, representation, layer

3.2 How do machines learn?

This section introduces neural-network learning as starting from randomized weights and adjusting them so predictions reflect patterns in data.

Key terms: weights, initialization, carefully scaled random values, predictions, patterns in the data

3.2.1 Learning from mistakes

This section explains how loss functions turn the gap between a neural network’s prediction and the correct answer into a measurable signal for learning.

Key terms: prediction, correct answer, loss, estimation error, y_pred, y_true, loss function, mean squared error, MSE, cross-entropy loss

3.2.2 Tracing the path of error

This section explains how backpropagation traces prediction error backward through a neural network to assign responsibility to weights, while introducing gradients, autodiff, and gradient instability in deep networks.

Key terms: backpropagation, loss, weights, error signal, derivative, gradient, automatic differentiation, autodiff, vanishing gradient problem, exploding gradients

3.2.3 Nudging in the right direction

This section explains how optimization converts error gradients into stable parameter updates through gradient descent and its practical refinements.

Key terms: optimization, gradient descent, gradient, loss, weights, biases, learning rate, hyperparameter, overshoot, optimizer

3.2.4 How rules emerge from examples

This section explains how neural networks extract general patterns through repeated training on many examples, using batches and epochs to organize and vary learning updates.

Key terms: instances, underlying pattern, regularities, batches, batch size, loss, updates, epochs, predictions, correct answers

3.2.5 What the network learns

This section explains that neural networks generalize learned data patterns to similar new inputs, but lack human-like understanding and can fail when real-world conditions differ from training data.

Key terms: training data, patterns, new inputs, loss, learned patterns, distribution shift, training conditions, learned regularities

3.3 Why learning is not memorizing

This section distinguishes genuine neural-network learning as generalizing patterns to new data from brittle memorization of training examples.

Key terms: learning, memorization, patterns, examples, representations, generalize, training experience, new data, reliable responses, patterns across data

3.3.1 When training becomes memorization

This section explains how overfitting turns apparent learning into brittle memorization of training examples and why models must be kept focused on patterns that generalize.

Key terms: overfitting, training data, training set, generalization, brittle, signal, noise, capacity, incidental quirks, correlations

3.3.2 Learning under constraint

This section explains how regularization constrains neural-network training so models generalize beyond training examples rather than relying on misleading details.

Key terms: regularization, generalization, overfitting, weight regularization, weights, smoother relationships, brittle behavior, dropout, neurons, pathway

3.3.3 Knowing when to stop

This section explains how validation data and early stopping identify overfitting and preserve a model that generalizes better to unseen data.

Key terms: validation set, training data, generalization, overfitting, early stopping, validation performance, test set, final model, training decisions

3.3.4 Augmenting the data

This section explains how data augmentation creates varied versions of existing training examples so models generalize beyond narrow, consistently presented data.

Key terms: data augmentation, generalization, training examples, transformed versions, underlying identity, underlying meaning, features that persist, exact presentation, structure, contexts

3.3.5 Intelligence in abstraction

This section explains neural-network abstraction as compressed pattern extension, contrasts it with human grounded conceptual abstraction, and frames the distinction as central to claims of machine intelligence.

Key terms: abstraction, compression, patterns, training data, representations, incidental correlations, human abstraction, grounding, invariants, conceptual understanding

3.4 One concept, many architectures

This section introduces why neural networks use different architectures to match the structural patterns of different kinds of data and tasks.

Key terms: neural networks, basic components, architecture, structure of real data, simple classification, space, time, compression, synthesis, reorganizing components

3.4.1 The limits of dense networks

This section explains why fully connected dense networks are flexible but inefficient for structured data, motivating architectures that encode spatial, temporal, or sequential structure.

Key terms: dense networks, fully connected networks, layers, neurons, dense layers, structure, spatial proximity, ordering, temporal rhythm, parameters

3.4.2 Convolutional networks: structure in space

This section explains how convolutional neural networks exploit local spatial structure in images through shared filters, pooling, and layered feature hierarchies.

Key terms: dense networks, flattened vector, adjacency, local structure, spatial relationships, convolutional neural networks (CNNs), filters, convolution kernels, convolution, shared weights

3.4.3 Recurrent networks: structure in sequences

This section explains how recurrent neural networks process ordered sequences by carrying a hidden state through time, while also showing why their memory breaks down over long spans.

Key terms: recurrent neural network, RNN, internal state, hidden state, sequential data, context, weights, shared weights, time steps, recurring patterns

3.4.4 LSTMs and GRUs: unlocking long-term memory

This section explains how LSTMs and GRUs use trainable gates to retain relevant information across long sequence gaps that basic recurrent networks struggle to handle.

Key terms: basic RNNs, hidden state, long gaps, signals fade, signals grow uncontrollably, Long Short-Term Memory networks, LSTMs, Gated Recurrent Units, GRUs, gates

3.4.5 Autoencoders: compressing the world

This section explains how autoencoders learn a compact latent representation of data by compressing an input and reconstructing it under a bottleneck constraint.

Key terms: autoencoder, encoder, decoder, latent space, reconstruction, bottleneck, features, underlying structure, noise, compression through description

3.4.6 Generative architectures: learning to create

This section explains how generative neural architectures learn the structure of data so they can synthesize new examples rather than only recognize, predict, or compress existing ones.

Key terms: generative architectures, data distribution, synthesize, structured internal representation, variational autoencoder, VAE, latent space, smooth, continuous, samples

3.4.7 Architecture as a form of intelligence

This section presents neural-network architecture as a hypothesis about information structure that shapes which patterns a machine can learn and what kinds of tasks it can handle.

Key terms: architecture, information flow, computational forms, regularities, training, convolutional networks, recurrent networks, LSTMs, GRUs, gating mechanisms

3.5 Specific approaches to language

This section introduces why language challenges general neural architectures and motivates adapting familiar components to handle linguistic structure across scales.

Key terms: order, context, meaning, signals far from where they’re used, general neural architectures, No single model, reshaping familiar components, different scales, organizes information

3.5.1 Why language challenges neural networks

This section explains why language exposes the limits of convolutional and recurrent neural networks and demands architectures that can preserve and revise context more globally.

Key terms: language modeling, tokens, distant dependencies, representations, convolutional models, recurrent models, evolving state, gated variants, long-range information, strictly sequential processing

3.5.2 Lessons from recurrence

This section explains how recurrent neural networks improved language modeling by carrying context forward token by token, while exposing the limits of storing all prior context in one evolving hidden state.

Key terms: recurrent networks, RNNs, token, hidden state, n-grams, next-word prediction, single evolving state, diffused information, entangled context, propagate

3.5.3 Pushing the memory barrier

This section explains how LSTMs and GRUs extended sequence models’ usable context while exposing the memory and parallelism limits of recurrent processing.

Key terms: LSTMs, GRUs, simple RNNs, context, dependencies, gating, sequence-to-sequence architectures, stacked LSTMs, neural machine translation, speech recognition

3.5.4 A word is worth a thousand pictures

This section explains how convolutional neural networks applied to text efficiently capture local linguistic patterns, while also showing both their usefulness and their limits for language tasks.

Key terms: convolutional networks, CNNs, filters, short-range patterns, local syntax, parallel architecture, local structure, explicit sequence modeling, recurrent models, dense models

3.5.5 Learning where to look

This section explains how attention lets sequence models consult the most relevant parts of an input on demand, overcoming the memory limits of recurrent and convolutional approaches and leading to Transformers.

Key terms: recurrent approaches, convolutional approaches, single evolving state, fixed local window, long-range dependencies, attention mechanisms, relevance weights, entire sequence, selectively retrieve, context on demand

3.5.6 Learn first, specialize later

This section explains how pretraining on broad raw text followed by task-specific transfer learning made language models more reusable, adaptable, and economical.

Key terms: pretraining, transfer learning, raw text, task-specific adaptation, labeled data, training example, expected answer, pretrained model, broadly pretrained model, applications

3.5.7 From models to true language

This section synthesizes why language modeling moved from sequential memory toward Transformer-style attention and pretraining to handle context at multiple scales.

Key terms: sequential models, single state, convolutional approaches, local structure, long-range dependencies, attention, directly accessible context, pretraining, structure, task

3.6 Summary

This summary consolidates how neural networks compute and learn, why architectures must match data structure, and how attention and pretraining advanced language modeling.

Where this is used later

Concepts explained in this chapter

← Chapter 2: Representing language Chapter 4: Large language models →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.