A music Transformer generates music one event at a time. Unlike MuseGAN, which treats a piece as a multidimensional object and generates a complete piece, it treats a performance as a sequence and predicts each next event from earlier ones. section 14.0
MIDI performances become sequences of four event types: note-on starts a pitch, note-off ends it, time-shift records elapsed time, and velocity records playing force. Each distinct event receives an integer index. There are 388 event tokens, plus end-of-sequence and padding tokens, for a vocabulary of 390. section 14.1 section 14.1.2
The model is decoder-only: it needs no encoder for a separate source language because it continues the same event sequence it receives. Token and learned position embeddings are added, then passed through six decoder blocks with causal self-attention. With 512-dimensional embeddings, eight attention heads, and sequences up to 2,048 indexes, the model outputs a logit for each vocabulary token at each position. section 14.1.2 section 14.3.2
Figure 14.1 The architecture of a music Transformer. Music files in MIDI formats are first converted into sequences of musical events. These events are then tokenized and converted into indexes. We organize these indexes into sequences of 2,048 elements, and each batch contains 2 such sequences. The input sequence first undergoes word embedding and positional encoding; the input embedding is the sum of these two components. This input embedding is then processed through six decoder layers, each utilizing self-attention mechanisms to capture the relationships among different musical events in the sequence. After passing through the decoder layers, the output undergoes layer normalization to ensure stability in the training process. It then passes through a linear layer, resulting in an output size of 390, which corresponds to the number of unique tokens in the vocabulary. This final output represents the predicted logits for the next musical event in the sequence.
Follow the diagram from MIDI events and indexes through embeddings, decoder layers, and the 390-token output to see where each representation changes. section 14.1.2
For training, tokenize piano performances, split the indexes into sequences of up to 2,048, and shift each target sequence one index ahead of its input. The model then learns next-event prediction by comparing its predictions with those targets using cross-entropy loss. The training run uses 100 epochs; validation performance can instead guide when to stop. section 14.1.3 section 14.4
Continue a piano prompt
Take the first 250 events of a test performance as a prompt. Repeatedly sample the next index, append it, and feed the longer sequence back until it reaches 1,000 indexes; then decode the result to a playable MIDI file. The sampling function applies softmax to temperature-scaled logits and samples from the 388 event tokens. Raising the temperature offers a way to change the output’s diversity. section 14.4.2 section 14.4
