liveBook · Manning

Sequence generation

As explained in Deep Learning with Python, Third Edition

The task of producing an ordered sequence of tokens, characters, or other elements from a model rather than a single label.

Sequence generation produces an ordered series of tokens, characters, or other elements instead of a single label. A language model can generate text by guessing the next token repeatedly. section 16.1

One task, several sequence types

The same general idea appears in different domains: LSTMs generated music, recurrent networks generated handwriting from pen-position time series, and language models generated text by predicting the next token. section 16.1 section 16.1

A practical milestone was GPT: unsupervised language-model pretraining produced a next-token predictor that could later be fine-tuned for text classification tasks. section 16.1

How Deep Learning with Python, Third Edition covers it

How deep it goes
Defines next-token prediction and surveys milestones, but stops short of implementation details or decoding procedures.
Examples
Uses music, handwriting trajectories, Shakespeare text generation, and GPT language pretraining as examples.
What you can do after
Evaluate how sequence-generation ideas extend from next-token prediction to generated music, handwriting, and text.
How it connects
Connects sequence generation with LSTM, recurrent neural networks, language models, Transformers, and fine-tuning.
Cover of Deep Learning with Python, Third Edition
liveBookmanning.com
Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 2 books explain sequence generation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.