A GPT model generates text with a decoder-only Transformer: information moves only from left to right, so each token uses left-context rather than future tokens. This simpler setup removes the encoder and lets pretraining use single, unpaired text sequences at large scale. section 16.2.1
One sequence for question and answer
To handle “Where is the capital of France?”, combine the question and its answer into one token sequence. Question and answer tokens share embeddings and parameters; the model predicts the continuation while attention prevents earlier tokens from using later ones. section 16.2.1
A practical mini-GPT implementation keeps the architecture focused:
- Remove cross-attention from each Transformer decoder block, leaving self-attention, feed-forward layers, normalization, and dropout. section 16.2.1
- Add token and positional embeddings so the model represents both token identity and sequence position. section 16.2.1
- Stack decoder layers and reverse the embedding projection to produce outputs over the vocabulary. section 16.2.1
Listing 16.4 A Transformer decoder block without cross-attention
from keras import layers class TransformerDecoder(keras.Layer): def __init__(self, hidden_dim, intermediate_dim, num_heads): super().__init__() key_dim = hidden_dim // num_heads self.self_attention = layers.MultiHeadAttention( #1 num_heads, key_dim, dropout=0.1 #1 ) #1 self.self_attention_layernorm = layers.LayerNormalization() #1 self.feed_forward_1 = layers.Dense(intermediate_dim, activation="relu")#2 self.feed_forward_2 = layers.Dense(hidden_dim) #2 self.feed_forward_layernorm = layers.LayerNormalization() #2 self.dropout = layers.Dropout(0.1) def call(self, inputs): residual = x = inputs #3 x = self.self_attention(query=x, key=x, value=x, use_causal_mask=True) #3 x = self.dropout(x) #3 x = x + residual #3 x = self.self_attention_layernorm(x) #3 residual = x #4 x = self.feed_forward_1(x) #4 x = self.feed_forward_2(x) #4 x = self.dropout(x) #4 x = x + residual #4 x = self.feed_forward_layernorm(x) #4 return x
Inspect the Keras decoder block that implements self-attention without cross-attention, with feed-forward processing, normalization, and dropout. section 16.2.1
Listing 16.6 Creating a mini-GPT functional model
keras.config.set_dtype_policy("mixed_float16") #1 vocab_size = tokenizer.vocabulary_size() hidden_dim = 512 intermediate_dim = 2056 num_heads = 8 num_layers = 8 inputs = keras.Input(shape=(None,), dtype="int32", name="inputs") embedding = PositionalEmbedding(sequence_length, vocab_size, hidden_dim) x = embedding(inputs) x = layers.LayerNormalization()(x) for i in range(num_layers): x = TransformerDecoder(hidden_dim, intermediate_dim, num_heads)(x) outputs = embedding(x, reverse=True) mini_gpt = keras.Model(inputs, outputs)
Inspect a complete mini-GPT assembled from positional embeddings, eight decoder layers, mixed-float16 computation, and a vocabulary projection. section 16.2.1
