liveBook · Manning

Transformer architecture

As explained in Transfer Learning for Natural Language Processing

A transformer is a neural network architecture that models sequences using attention mechanisms rather than recurrent connections.

A transformer models sequences using attention rather than recurrent connections. In machine translation, an encoder represents the input and a decoder produces the output; attention helps connect relevant parts of the input and output. Self-attention extends this idea to relationships among parts of the same sequence, building context for each text segment. section 1.3.1 section 7.1

Unlike a recurrent network, a transformer does not depend on the previous position’s hidden state to process the next position. This makes sequence processing more amenable to parallel hardware, although attention still models dependencies between tokens. Large transformer models can nevertheless require substantial computing resources. section 7.1 section 7.0

Figure 7.1 A high-level representation of the transformer architecture, showing stacked encoders, decoders, input/output embeddings, and positional encodings
07_01

The original encoder-decoder design combines stacked encoders and decoders with embeddings and positional encodings. section 7.1

Figure 7.2 A simplified decomposition of the encoder and decoder into self-attention, encoder-decoder attention, and feedforward neural networks
07_02

The encoder and decoder contain self-attention and feedforward networks; the decoder also uses encoder-decoder attention. section 7.1

Context in a sentence

Consider: “He didn’t want to talk about cells on the cell phone because he considered it boring.” Self-attention can learn relationships among the sentence’s parts, helping build context around occurrences of “cells” and “cell.” section 7.1

Transformer-based models can be adapted to different NLP tasks. BERT uses transformer encoders to build context and can be used for classification; GPT uses a causal modeling objective suited to text generation. Multilingual BERT extends BERT to multiple languages. section 3.2.2 section 7.3 section 8.3

Listing 3.8 Converting data to form expected by BERT, building and training model
def build_model(max_seq_length):                                           #1
    in_id = tf.keras.layers.Input(shape=(max_seq_length,), name="input_ids")
    in_mask = tf.keras.layers.Input(shape=(max_seq_length,), name="input_masks")
    in_segment = tf.keras.layers.Input(shape=(max_seq_length,), name="segment_ids")
    bert_inputs = [in_id, in_mask, in_segment]
    bert_output = BertLayer(n_fine_tune_layers=0)(bert_inputs)             #2
    dense = tf.keras.layers.Dense(256, activation="relu")(bert_output)
    pred = tf.keras.layers.Dense(1, activation="sigmoid")(dense)
 
    model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
    model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
    model.summary()
 
    return model
 
 
def initialize_vars(sess):                                                 #3
    sess.run(tf.local_variables_initializer())
    sess.run(tf.global_variables_initializer())
    sess.run(tf.tables_initializer())
    K.set_session(sess)
 
 
 
bert_path = "https:/ /tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1"
tokenizer = create_tokenizer_from_hub_module(bert_path)                    #4
 
train_examples = convert_text_to_examples(train_x, train_y)                #5
test_examples = convert_text_to_examples(test_x, test_y)
 
# Convert to features
(train_input_ids,train_input_masks,train_segment_ids,train_labels) =       #6
     convert_examples_to_features(tokenizer, train_examples,               #6
     max_seq_length=maxtokens)                                             #6
(test_input_ids,test_input_masks,test_segment_ids,test_labels) = 
     convert_examples_to_features(tokenizer, test_examples,  
     max_seq_length=maxtokens)
 
model = build_model(maxtokens)                                             #7
 
initialize_vars(sess)                                                      #8
 
history = model.fit([train_input_ids, train_input_masks, train_segment_ids],#9
train_labels,validation_data=([test_input_ids, test_input_masks, 
test_segment_ids],test_labels), epochs=5, batch_size=32)

A classification model uses a BERT layer with token IDs, masks, and segment IDs, followed by dense prediction layers. section 3.2.2

How Transfer Learning for Natural Language Processing covers it

How deep it goes
Explains self-attention, encoder-decoder components, and parallelization; defers a complete BERT architectural discussion in the baseline treatment.
Examples
Uses machine translation, a sentence about cells and a cell phone, and a BERT classifier to illustrate transformer applications.
What you can do after
Evaluate and understand transformer components and their parallelization advantages, and apply pretrained BERT through Keras and TensorFlow Hub.
How it connects
Connects self-attention with encoder-decoder models, recurrent networks, transfer learning, BERT, and GPT.

Also explained in

Compare how 29 books explain transformer architecture →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Transfer Learning for Natural Language Processing. The section links point to the parts of the book it draws on.