Model architecture is the arrangement of computational layers, connections, inputs, and outputs that gives a neural network its structure. Choosing one is problem-dependent: designers balance what the model must learn against available memory, computing power, response time, or visual awareness. section 9.11 section 11.11
When shaping an architecture, consider:
- Layer count and arrangement, which determine how much structure the network can represent. section 9.11
- Input and output design, which must match the prediction or generation problem. section 9.11
- Width, depth, attention heads, context window, and feed-forward size in Transformer-based language models. section 11.11
- Memory capacity, compute cost, and latency for language models, or spatial awareness, multiscale processing, and parameter efficiency for image generation. section 11.11 section 12.12
From a small ANN to a specialized model
A simple prediction network can gain another hidden layer and a second output when the problem requires a different structure. A language model instead chooses dimensions such as width, depth, and attention heads, while an image-generation model must preserve spatial relationships and combine broad composition with fine detail. section 9.11 section 11.11 section 12.12
Figure 11.22 Comparison of different ANN architectures aimed at training LLMs
Compare ANN architectures aimed at training language models and connect structural choices with their tradeoffs. section 11.11
