In music Transformer training, y is the target sequence: for each input token in x, it supplies the next music-token index the model should predict. It comes from the same music piece as x, but starts one position later. For an illustrative indexed sequence [324, 366, 67, 60], x could be [324, 366, 67] and y would be [366, 67, 60]. section 14.1.3 section 14.2.3
To prepare each training pair:
- Convert MIDI music into event-token indexes, then make x and y sequences of 2,048 indexes each. This length is intended to retain long-range musical relationships while keeping training manageable. section 14.1.3 section 14.2.3
- For a piece longer than 2,048 indexes, take the first 2,048 for x and indexes 2 through 2,049 for y. section 14.2.3
- For a piece of at most 2,048 indexes, put its following tokens in y, mark its end with index 388, and fill unused positions with padding index 389. The 388 event tokens occupy indexes 0–387. section 14.2.3
Listing 14.5 Creating training data
import torch,os,pickle
max_seq=2048
def create_xys(folder):
files=[os.path.join(folder,f) for f in os.listdir(folder)]
xys=[]
for f in files:
with open(f,"rb") as fb:
music=pickle.load(fb)
music=torch.LongTensor(music)
x=torch.full((max_seq,),389, dtype=torch.long)
y=torch.full((max_seq,),389, dtype=torch.long) #1
length=len(music)
if length<=max_seq:
print(length)
x[:length]=music #2
y[:length-1]=music[1:] #3
y[length-1]=388 #4
else:
x=music[:max_seq]
y=music[1:max_seq+1]
xys.append((x,y))
return xys
The create_xys() listing shows both cases: music[1:max_seq+1] supplies long-piece targets, while shorter pieces receive an end index and padding. section 14.2.3
During training, the Transformer receives x and predicts the next token at each position. Its predictions are compared with y using cross-entropy loss. The shift turns an indexed performance into many next-event prediction targets. section 14.1.3
