A training dataset is the collection of examples a model learns from. Its contents shape what the model can produce: a GAN trained on clothing images generates clothing-like images, while one trained on faces generates face-like images. For text generation, the writing style likewise depends on the training text. section 1.2.1 section 12.1.2
For a GPT text generator, dataset choice is a balance between length and variety. Too little variety can lead to passages copied from training text. A sufficiently large collection helps the model learn a style, but an overly long collection demands more computation. Training for too many epochs can also produce text identical to training passages; dataset choice alone does not prevent overfitting. section 12.1.2 section 12.5
Preparing a small writing-style dataset
To generate Hemingway-style text, combine three novels rather than using just one. Split the text into word and punctuation tokens, assign each unique token an integer, and divide the resulting sequence into inputs of up to 128 indexes. Shift each input sequence one token to form its target: the model learns to predict the next token from preceding tokens. One novel would offer less variety; too many would enlarge the vocabulary and make short training runs harder. section 12.1.2 section 12.1
Figure 12.2 The training process for a decoder-only Transformer to generate text, Hemingway-style.
Follow the diagram from training text through numerical sequences to next-token prediction. section 12.1.2
A training set is not a validation set. In a clothing-image classification example, 60,000 training observations become 50,000 for training and 10,000 for validation. Validation performance helps choose settings such as the number of epochs without judging the model only on examples it trains on. section 2.4.1
The preparation changes with the data. Images become tensors; text becomes token indexes; MIDI performances become indexed musical events. In each case, the examples must be converted into a form the network can receive before training begins. section 2.2.1 section 14.1.3
