Sequence generation produces ordered data elements with a computational model. Applications have included music, handwriting trajectories, and text generation by predicting the next token. section 16.1
Representative developments include:
- LSTM models generated music and later supported broader generative recurrent-network research. section 16.1
- Recurrent mixture density networks generated human-like handwriting from time series of pen positions. section 16.1
- Language models generated text by guessing the next token; GPT combined this pretraining approach with the Transformer architecture and later fine-tuning. section 16.1
Next-token text generation
A model receives preceding text and predicts the next token, repeating the process to produce a sequence. This approach underlies the described GPT pretraining setup. section 16.1
