Model depth is the number of successive layers that transform a representation. Deeper networks can build features step by step, combining simpler cues into more abstract ones; depth and width both contribute to a model’s capacity, but in different ways. section 4.2.1 section 3.1.2
Greater depth also creates training challenges: repeated transformations can distort or weaken useful information and make learning unstable. Residual connections preserve earlier representations through incremental updates, while normalization keeps values within a more predictable range. Normalization placement varies: the original Transformer applied it after each layer, while many modern models apply it beforehand. section 4.1.5
Preserving a useful relationship
After attention links “rises” with “sun” and “east,” a later layer may focus on other sentence details. A residual connection helps keep the earlier links available rather than replacing them with the newer transformation. section 4.1.5
Figure 3.2 A deep neural network as a sequence of layers. The input layer receives raw data, the hidden layers transform it through learned representations, and the output layer produces the model’s prediction. In this example, the four input nodes might represent measurable features of an email, such as word frequencies or metadata, while the three output nodes might represent possible categories, such as personal, work-related, or spam. The network transforms these input features into output probabilities through the connections learned in the hidden layers.

Follow how hidden layers transform input features into progressively learned representations and then an output. section 3.1.2





