mini-GPT is a small generative language model trained to predict the next token in text. Its practical pipeline downloads a portion of C4, tokenizes documents with SentencePiece, and prepares fixed-length sequences for Transformer pretraining. section 16.2 section 16.2.1
A complete mini-GPT workflow combines:
- Dataset preparation: extract text shards, restore escaped newlines, tokenize documents, append an end-of-text token, and create shifted input-target sequences. section 16.2
- Token representation: embed tokens and positions, then reuse the token-embedding weights as a reverse projection from hidden states to vocabulary predictions. section 16.2.1
- Decoder stack: pass representations through eight decoder layers using self-attention, feed-forward transformations, normalization, residual connections, and dropout. section 16.2.1
- Training configuration: use mixed-precision computation for a 41-million-parameter model whose scale makes training substantially compute-intensive. section 16.2 section 16.2.1
Concrete configuration
The functional model accepts integer token sequences, uses a 512-dimensional hidden representation, eight attention heads, an intermediate dimension of 2056, and eight Transformer decoder layers before projecting back into vocabulary space. section 16.2.1
Listing 16.6 Creating a mini-GPT functional model
config_set_dtype_policy("mixed_float16") #1
vocab_size <- tokenizer$vocabulary_size()
hidden_dim <- 512
intermediate_dim <- 2056
num_heads <- 8
num_layers <- 8
inputs <- keras_input(shape = c(NA), dtype = "int32", name = "inputs")
embedding <-
layer_positional_embedding(, sequence_length, vocab_size, hidden_dim)
x <- inputs |>
embedding() |>
layer_layer_normalization()
for (i in seq_len(num_layers)) {
x <- x |>
layer_transformer_decoder(hidden_dim, intermediate_dim, num_heads)
}
outputs <- x |> embedding(reverse = TRUE)
mini_gpt <- keras_model(inputs, outputs)
Inspect the complete R/Keras model definition, including mixed precision, positional embeddings, decoder stacking, and the reverse embedding projection. section 16.2.1
