In music-token training, x is the input sequence of integer indexes fed to a Transformer. Each index represents a musical event. The matching target, y, is the same sequence shifted one position forward, so the model learns to predict the next event from the events it has seen. section 14.1.3
To make an (x, y) pair from an indexed music piece:
- For a long piece, take its first 2,048 indexes as x and indexes 2 through 2,049 as y. section 14.2.3
- For a piece of 2,048 indexes or fewer, fill the unused positions with padding index 389. Put end-of-sequence index 388 after the last predicted event in y. section 14.2.3
A short piece
If a piece has 15 event indexes, x contains those 15 indexes followed by padding. The first 14 positions of y contain the following events; its next position marks the end, and the rest is padding. Both tensors still have shape [2048]. section 14.2.3
Listing 14.5 Creating training data
import torch,os,pickle
max_seq=2048
def create_xys(folder):
files=[os.path.join(folder,f) for f in os.listdir(folder)]
xys=[]
for f in files:
with open(f,"rb") as fb:
music=pickle.load(fb)
music=torch.LongTensor(music)
x=torch.full((max_seq,),389, dtype=torch.long)
y=torch.full((max_seq,),389, dtype=torch.long) #1
length=len(music)
if length<=max_seq:
print(length)
x[:length]=music #2
y[:length-1]=music[1:] #3
y[length-1]=388 #4
else:
x=music[:max_seq]
y=music[1:max_seq+1]
xys.append((x,y))
return xys
The create_xys() listing shows the two slicing cases and the separate indexes used for padding and sequence end. section 14.2.3
