liveBook · Manning

Depth

As explained in Machines that Think

Model depth is the number of successive computational layers or transformation stages in a model.

Model depth is the number of successive layers that transform a representation. Deeper networks can build features step by step, combining simpler cues into more abstract ones; depth and width both contribute to a model’s capacity, but in different ways. section 4.2.1 section 3.1.2

Greater depth also creates training challenges: repeated transformations can distort or weaken useful information and make learning unstable. Residual connections preserve earlier representations through incremental updates, while normalization keeps values within a more predictable range. Normalization placement varies: the original Transformer applied it after each layer, while many modern models apply it beforehand. section 4.1.5

Preserving a useful relationship

After attention links “rises” with “sun” and “east,” a later layer may focus on other sentence details. A residual connection helps keep the earlier links available rather than replacing them with the newer transformation. section 4.1.5

Figure 3.2 A deep neural network as a sequence of layers. The input layer receives raw data, the hidden layers transform it through learned representations, and the output layer produces the model’s prediction. In this example, the four input nodes might represent measurable features of an email, such as word frequencies or metadata, while the three output nodes might represent possible categories, such as personal, work-related, or spam. The network transforms these input features into output probabilities through the connections learned in the hidden layers.

Follow how hidden layers transform input features into progressively learned representations and then an output. section 3.1.2

How Machines that Think covers it

How deep it goes
Defines depth through successive transformations and connects it to representation building and training instability, without formal measurement detail.
Examples
Uses image features that progress from edges to shapes, sentence relationships around “rises,” and tasks such as face recognition and sentiment detection.
What you can do after
Evaluate how greater depth can expand representational capacity while increasing training challenges.
How it connects
Ties depth to hidden layers, deep learning, residual connections, normalization, attention, and feedforward layers.

Also explained in

Compare how 6 books explain depth →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.