liveBook · Manning

Transformer architecture

As explained in Domain-Specific Small Language Models

A neural network architecture for sequence modeling built around self-attention and parallel processing, underlying both large and small language models.

Transformers model sequences with self-attention, relating positions across an input to build its representation. Unlike recurrent neural networks, they process the input all at once without a recurrent structure, enabling greater parallelism and faster training. Word embeddings represent words as vectors that capture semantic and syntactic properties. section 1.2 section 1.7

Common architecture choices adapt the Transformer to different tasks:

  • BERT uses the encoder for classification and prediction; GPT uses the decoder for text generation. section 1.7
  • Pretrained models can be adapted to specialized domains through transfer learning or fine-tuning. section 1.7

Inference with GPT-Neo

GPT-Neo large, a 2.7-billion-parameter model that replicates the GPT-3 architecture, serves as a reference for content generation. Its inference illustrates factors affecting execution speed, accuracy, and compute cost. section 4.1

How Domain-Specific Small Language Models covers it

How deep it goes
Defines self-attention and explains its parallelism advantage over recurrent networks, while leaving multimodal applications out of scope.
Examples
Contrasts sequential RNN processing with whole-sequence Transformers and uses GPT-Neo large for content-generation inference.
What you can do after
Apply it via Hugging Face Transformers and Accelerate to run generation while considering speed, accuracy, and compute cost.
How it connects
Connects self-attention and word embeddings to RNN, LSTM, and GRU contrasts, then distinguishes encoder-only BERT from decoder-only GPT.

Also explained in

Compare how 29 books explain transformer architecture →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.