A neural network layer is a compositional processing stage within a model. Transformer encoders and decoders form layers that process representations, and multiple layers can be stacked to build more capable models. section 2.3
In the transformer architecture, layers are used to:
- Process input text into hidden states that represent its meaning. section 2.3
- Pass representations through stacked processing stages before generating output text. section 2.3
- Scale model architecture by combining multiple layers, including decoder-only designs such as GPT-3. section 2.3
Transformer depth
GPT-3 is described as a decoder-only transformer with 96 layers, illustrating how a model can stack many processing stages rather than relying on one computational step. section 2.3
A neural network layer is narrower than the model layer in a GenAI application architecture. The model layer manages access to foundational, fine-tuned, and small language models through a consistent interface. section 10.2.3 section 10.5
