A transformer models sequences using attention rather than recurrent connections. In machine translation, an encoder represents the input and a decoder produces the output; attention helps connect relevant parts of the input and output. Self-attention extends this idea to relationships among parts of the same sequence, building context for each text segment. section 1.3.1 section 7.1
Unlike a recurrent network, a transformer does not depend on the previous position’s hidden state to process the next position. This makes sequence processing more amenable to parallel hardware, although attention still models dependencies between tokens. Large transformer models can nevertheless require substantial computing resources. section 7.1 section 7.0
Figure 7.1 A high-level representation of the transformer architecture, showing stacked encoders, decoders, input/output embeddings, and positional encodings
The original encoder-decoder design combines stacked encoders and decoders with embeddings and positional encodings. section 7.1
Figure 7.2 A simplified decomposition of the encoder and decoder into self-attention, encoder-decoder attention, and feedforward neural networks
The encoder and decoder contain self-attention and feedforward networks; the decoder also uses encoder-decoder attention. section 7.1
Context in a sentence
Consider: “He didn’t want to talk about cells on the cell phone because he considered it boring.” Self-attention can learn relationships among the sentence’s parts, helping build context around occurrences of “cells” and “cell.” section 7.1
Transformer-based models can be adapted to different NLP tasks. BERT uses transformer encoders to build context and can be used for classification; GPT uses a causal modeling objective suited to text generation. Multilingual BERT extends BERT to multiple languages. section 3.2.2 section 7.3 section 8.3
Listing 3.8 Converting data to form expected by BERT, building and training model
def build_model(max_seq_length): #1
in_id = tf.keras.layers.Input(shape=(max_seq_length,), name="input_ids")
in_mask = tf.keras.layers.Input(shape=(max_seq_length,), name="input_masks")
in_segment = tf.keras.layers.Input(shape=(max_seq_length,), name="segment_ids")
bert_inputs = [in_id, in_mask, in_segment]
bert_output = BertLayer(n_fine_tune_layers=0)(bert_inputs) #2
dense = tf.keras.layers.Dense(256, activation="relu")(bert_output)
pred = tf.keras.layers.Dense(1, activation="sigmoid")(dense)
model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
model.summary()
return model
def initialize_vars(sess): #3
sess.run(tf.local_variables_initializer())
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())
K.set_session(sess)
bert_path = "https:/ /tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1"
tokenizer = create_tokenizer_from_hub_module(bert_path) #4
train_examples = convert_text_to_examples(train_x, train_y) #5
test_examples = convert_text_to_examples(test_x, test_y)
# Convert to features
(train_input_ids,train_input_masks,train_segment_ids,train_labels) = #6
convert_examples_to_features(tokenizer, train_examples, #6
max_seq_length=maxtokens) #6
(test_input_ids,test_input_masks,test_segment_ids,test_labels) =
convert_examples_to_features(tokenizer, test_examples,
max_seq_length=maxtokens)
model = build_model(maxtokens) #7
initialize_vars(sess) #8
history = model.fit([train_input_ids, train_input_masks, train_segment_ids],#9
train_labels,validation_data=([test_input_ids, test_input_masks,
test_segment_ids],test_labels), epochs=5, batch_size=32)
A classification model uses a BERT layer with token IDs, masks, and segment IDs, followed by dense prediction layers. section 3.2.2




























