How deeply does Sutskever's List cover encoder-decoder architecture?

Explains recurrent encoding, fixed-vector conditioning, and decoding, emphasizing how compression strains longer inputs.

Encoder-decoder architecture

As explained in Sutskever's List

A sequence-to-sequence model maps an input sequence to an output sequence using learned encoder and decoder components or an equivalent transformation.

An encoder–decoder maps a variable-length input sequence to a variable-length output. In the 2014 sequence-to-sequence design, an LSTM encoder compressed the input into one fixed-size “thought vector”; a decoder used it to generate the output one word at a time. This enabled machine translation, but forced short and long sentences alike into the same-size representation. section 5.3

The fixed-vector bottleneck becomes more strained as sentences grow. Pointer Networks show a related encoder–decoder can instead use attention to select positions in the input, useful when outputs must come from the input rather than a fixed vocabulary. section 5.3 section 5.4

How Sutskever's List covers it

liveBook manning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionA neural architecture that encodes an input sequence into a representation and decodes that representation into an output sequence.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA neural sequence transduction model that maps an input sequence to an output sequence, potentially of different length, using an encoder and decoder.
liveBook manning.com

Deep Learning with Python, Third EditionA model that maps an input sequence to an output sequence, often used for translation or generation tasks.
liveBook manning.com

Knowledge Graphs and LLMs in ActionAn encoder-decoder architecture is a model design in which an encoder transforms input data into a learned representation and a decoder transforms that representation into predictions or reconstructed outputs.
liveBook manning.com

Deep Learning with R, Third EditionA sequence-to-sequence model predicts a target sequence from a separate source sequence, typically conditioning each target token on prior target tokens.
liveBook manning.com

Optimization Algorithms: AI techniques for design, planning, and control problemsA neural model that transforms an input sequence into an output sequence, typically allowing the two sequences to have different lengths.
liveBook manning.com

Causal AIA sequence-to-sequence model maps an input sequence to an output sequence, often for tasks such as translation, summarization, or conditional generation.
liveBook manning.com

Learn Generative AI with PyTorch
liveBook manning.com

Effective Conversational AIAn encoder-decoder architecture is a language-model architecture that encodes an input sequence and decodes a separate output sequence.

Compare how 10 books explain encoder-decoder architecture →

Related concepts

BARTincludes · BART is a denoising sequence-to-sequence Transformer model with a bidirectional encoder and an autoregressive decoder.
decodehas part · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
link predictionuses · Link prediction is a graph-learning task that estimates whether a connection exists or may occur between nodes, often by scoring candidate links.
retrieval-augmented generationused by · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.