Sequence prediction trains a model to predict the next element from earlier elements in an ordered sequence. In text, the elements are often tokens, so the model learns how word order and context shape what comes next. section 8.0 section 8.1.1
A basic text-prediction pipeline works as follows:
- Tokenize text into individual words or other tokens. section 8.0
- Map each token to an integer index and arrange the indexes into sequences of a chosen length. section 8.0
- Shift the input sequence by one token to create the target sequence. section 8.0
- Generate text iteratively by appending each predicted token to the growing prompt. section 8.0
Recurrent models such as LSTMs process sequence information step by step, while Transformers use attention to relate elements across long distances and support parallel training. Both approaches address sequence-to-sequence prediction, but Transformers are better suited to large-scale text modeling. section 9.0 section 8.0
Generating a sentence
Given the prompt “Anna and the,” an LSTM predicts the most likely next token, appends it to the prompt, and repeats the process until the desired length is reached. Temperature changes prediction randomness, while top-K sampling restricts choices to the most probable tokens. section 8.0
