A decoder-only transformer generates a sequence from tokens it already has. It has a stack of decoder layers but no encoder stack: given a prompt, it predicts a likely next token, appends it, and repeats. GPT-2 uses this design for text generation. section 9.1.3 section 11.0
Contrast this with an encoder–decoder translator. For English-to-French translation, an encoder represents the English input and passes that representation to a decoder. A decoder-only text generator instead works from preceding tokens in the same sequence; a music generator can likewise continue a sequence of musical events. section 11.0 section 14.1.2
A music-generation model makes the pipeline concrete:
- Convert MIDI music into event-token indexes. Add learned token and position embeddings to represent each event and its place in the sequence. section 14.1.2
- Pass those embeddings through six decoder layers. Causal self-attention uses eight heads to capture relationships among earlier musical events. section 14.1.2
- Apply layer normalization and a linear layer to produce logits for 390 possible tokens; use softmax to obtain probabilities when generating the next event. section 14.1.2 section 14.3.2
Figure 14.1 The architecture of a music Transformer. Music files in MIDI formats are first converted into sequences of musical events. These events are then tokenized and converted into indexes. We organize these indexes into sequences of 2,048 elements, and each batch contains 2 such sequences. The input sequence first undergoes word embedding and positional encoding; the input embedding is the sum of these two components. This input embedding is then processed through six decoder layers, each utilizing self-attention mechanisms to capture the relationships among different musical events in the sequence. After passing through the decoder layers, the output undergoes layer normalization to ensure stability in the training process. It then passes through a linear layer, resulting in an output size of 390, which corresponds to the number of unique tokens in the vocabulary. This final output represents the predicted logits for the next musical event in the sequence.
The diagram follows MIDI event indexes through embeddings, six decoder layers, and next-event logits. section 14.1.2
The architecture also scales to different model sizes. The music model uses 512-dimensional embeddings and about 20.16 million parameters; GPT-2XL has over 1.5 billion. A GPT-2XL implementation can load pretrained weights from Hugging Face, then sample next tokens from a prompt. The takeaway is the same in either application: choose tokens and model size for the task, while keeping generation grounded in preceding tokens. section 14.1.2 section 14.3.2 section 11.0
