A GPT model is an autoregressive Transformer language model that generates text by predicting the next token from preceding context. Its central idea is simple: pretrain one language model on large amounts of text, then reuse it for generation or downstream tasks such as text classification and question answering. section 16.1
A mini-GPT architecture
A compact implementation accepts integer token sequences, combines token and positional embeddings, applies layer normalization, then passes the result through eight decoder-only Transformer layers. Each layer uses eight-headed self-attention, a feed-forward network, residual connections, normalization, and dropout. A reversed embedding projection produces token predictions. In R with Keras, the functional model follows this shape: inputs → positional embedding → normalization → repeated decoder blocks → reversed embedding output. section 16.2.1
GPT’s design makes several practical trade-offs:
- Removes the encoder and cross-attention from the sequence-to-sequence Transformer, allowing information to flow only from left to right. section 16.2.1
- Feeds questions and answers as one sequence, embedding both token types in the same latent space with shared parameters. section 16.2.1
- Trades some expressivity for simpler pretraining, because arbitrary internet text can provide single-sequence training data without paired inputs and outputs. section 16.2.1
- Uses dropout inside each decoder layer when stacking many layers, helping prevent overfitting. section 16.2.1
- Shows scale as a major source of capability: GPT-1, GPT-2, and GPT-3 grew from 117 million to 175 billion parameters. section 16.1
Listing 16.4 Transformer decoder block without cross-attention
layer_transformer_decoder <- new_layer_class(
"TransformerDecoder",
initialize = function(hidden_dim, intermediate_dim, num_heads) {
super$initialize()
key_dim <- hidden_dim %/% num_heads
self$self_attention <- layer_multi_head_attention(
num_heads = num_heads,
key_dim = key_dim,
dropout = 0.1
)
self$self_attention_layernorm <- layer_layer_normalization() #1
self$feed_forward_1 <- layer_dense(units = intermediate_dim, #1
activation = "relu") #2
self$feed_forward_2 <- layer_dense(units = hidden_dim) #2
self$feed_forward_layernorm <- layer_layer_normalization() #2
self$dropout <- layer_dropout(rate = 0.1) #2
},
call = function(inputs) {
residual <- x <- inputs #3
x <- self$self_attention(query = x, key = x, value = x, #3
use_causal_mask = TRUE) #3
x <- x |> self$dropout() #3
x <- x + residual #3
x <- x |> self$self_attention_layernorm() #3
residual <- x #4
x <- x |> #4
self$feed_forward_1() |> #4
self$feed_forward_2() |> #4
self$dropout() #4
x <- x + residual #4
x <- x |> self$feed_forward_layernorm()
x
}
)
Inspect the decoder block that implements masked self-attention, feed-forward processing, normalization, residual structure, and dropout without cross-attention. section 16.2.1
Listing 16.6 Creating a mini-GPT functional model
config_set_dtype_policy("mixed_float16") #1
vocab_size <- tokenizer$vocabulary_size()
hidden_dim <- 512
intermediate_dim <- 2056
num_heads <- 8
num_layers <- 8
inputs <- keras_input(shape = c(NA), dtype = "int32", name = "inputs")
embedding <-
layer_positional_embedding(, sequence_length, vocab_size, hidden_dim)
x <- inputs |>
embedding() |>
layer_layer_normalization()
for (i in seq_len(num_layers)) {
x <- x |>
layer_transformer_decoder(hidden_dim, intermediate_dim, num_heads)
}
outputs <- x |> embedding(reverse = TRUE)
mini_gpt <- keras_model(inputs, outputs)
Inspect the complete mini-GPT assembly, including mixed-precision configuration, dimensions, repeated decoder layers, and output projection. section 16.2.1
