liveBook · Manning

Transformer architecture

As explained in Fight Fraud with Machine Learning

A transformer is a neural-network architecture that models relationships in sequences primarily through attention mechanisms rather than recurrence.

A transformer is a deep-learning architecture built entirely around attention layers, without recurrent connections. For a sequence, self-attention assigns coefficients to its elements, indicating how much each element should influence predicted outputs. Multiple self-attention layers form multi-head attention; their concatenated features let the model attend to different parts of a sequence in different proportions. section 4.3.4 section 4.3.4

Figure 4.19 Transformer model architecture containing multi-head attention layers that use attention mechanism only to extract complex information from input sequence.

The architecture diagram highlights repeated multi-head attention layers as the mechanism used to extract complex information from an input sequence. section 4.3.4

Attention replaced the earlier pattern of adding an attention layer after a recurrent layer: rather than carry information through recurrent time-step connections, a transformer uses attention alone. Repeating multi-head attention layers produces increasingly rich representations for complex prediction problems. section 4.3.4

Translation with an encoder and decoder

For English-to-French translation, an encoder receives an English sentence after its words are converted into numeric token vectors. It passes the resulting representation to a decoder, which generates the corresponding French sentence. This encoder–decoder arrangement is a simplified transformer architecture for translation. section 5.2.1 section 5.2.1

Transformers are deployed across fraud-focused tasks and data types:

  • Text-based phishing detection can use GPT, a transformer trained on large volumes of text for next-word language modeling. section 5.0
  • Vision transformers support visual tasks such as extracting information from identity documents during KYC automation. section 4.4
  • Transformer models can classify processed audio as real or fake for voice-faking detection. section 8.0 section 8.2
Figure 8.7 Steps in building, training, validating, and testing a transformer-based voice faking detection model.

The workflow connects a transformer classifier to practical training: define layers and forward pass, set hyperparameters, train and validate, then test on untouched data. section 8.2

How Fight Fraud with Machine Learning covers it

How deep it goes
Explains self-attention coefficients, multi-head concatenation, and encoder–decoder translation, then outlines Torch training without detailing layer internals.
Examples
Uses English-to-French translation, GPT phishing detection, KYC identity documents, and real-versus-fake voice classification.
What it assumes
Introduces tokenization from words to numeric vectors before explaining encoder–decoder translation.
What you can do after
Build it from scratch as a Torch classifier by defining layers and forward pass, then training, validating, and testing it.
How it connects
Connects transformers with self-attention, recurrent neural networks, tokenization, GPT, large language models, and vision transformers.

Also explained in

Compare how 40 books explain transformer architecture →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.