Transformers model sequences with self-attention, relating positions across an input to build its representation. Unlike recurrent neural networks, they process the input all at once without a recurrent structure, enabling greater parallelism and faster training. Word embeddings represent words as vectors that capture semantic and syntactic properties. section 1.2 section 1.7
Common architecture choices adapt the Transformer to different tasks:
- BERT uses the encoder for classification and prediction; GPT uses the decoder for text generation. section 1.7
- Pretrained models can be adapted to specialized domains through transfer learning or fine-tuning. section 1.7
Inference with GPT-Neo
GPT-Neo large, a 2.7-billion-parameter model that replicates the GPT-3 architecture, serves as a reference for content generation. Its inference illustrates factors affecting execution speed, accuracy, and compute cost. section 4.1




























