A source sequence is the fixed input to a sequence-to-sequence model: its tokens become vectors that an encoder processes before a decoder predicts the target sequence. section 15.2
Translation input
In machine translation, the source sequence is the text to translate. The encoder converts it into an intermediate representation, which the decoder uses with a start token to generate target tokens until an end token appears. section 15.2
Attention treats the encoded source sequence as a set of vectors. For each target token, it scores every source vector, normalizes the scores with softmax, and forms a weighted combination. Dot-product attention compares target and source vectors; parameterized and multi-head variants transform them before combining information. section 15.3.1
A source sequence participates in prediction through this practical flow:
- Encode the source sequence into vectors. section 15.2
- Score source vectors according to their relevance to the current target token. section 15.3.1
- Combine the vectors with normalized attention weights. section 15.3.1
Figure 15.1 Sequence-to-sequence learning: the source sequence is processed by the encoder and is then sent to the decoder. The decoder looks at the target sequence so far and predicts the target sequence offset by one step in the future. During inference, we generate one target token at a time and feed it back into the decoder.
Trace how the encoder processes the source sequence and passes it to a decoder that generates the target sequence one token at a time. section 15.2
