Teacher forcing

As explained in Inside Deep Learning: Math, Algorithms, Models

A training approach for sequence decoders that feeds the true previous token as the next input instead of the model's own prediction.

Teacher forcing trains a sequence decoder by feeding the true previous token as the next input instead of the model’s own prediction. That makes decoding easier during training and helps the model learn sequences faster. section 11.3.2 section 11.3.3

In Seq2Seq decoding, the encoder’s last hidden state initializes the decoder, then each step chooses between autoregressive input and teacher forcing. The target sequence must be available during training because the decoder needs the correct next token. section 11.3 section 11.4.1

Typical behavior in this implementation:

• Use the ground-truth token at step t as input for step t + 1 during training. section 11.3.2
• Use autoregressive sampling when teacher forcing is not selected or when predicting new data. section 11.3.3
• Keep the model in autoregressive mode at evaluation time because the answer is unknown. section 11.3.3

Decoder step selection

The forward pass checks training mode, then either sets next_words from target[:, t] or samples from the current prediction; at test time it uses argmax. A figure shows the decoder ignoring its own prediction and feeding the correct previous output instead. section 11.4.1

Figure 11.5 Example of teacher forcing. For the decoder, we ignore the predictions and at step t, we feed in the correct previous output y_{t − 1}. The predictions are still used when the loss is computed.

Shows the decoder input being replaced by the correct prior token while predictions still contribute to the loss. section 11.3.2

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Explains the input swap, its training benefit, and the training-versus-evaluation control flow in code.

Examples

Uses a decoder-step figure and a forward-pass code path that selects the next decoder input.

What it assumes

Assumes basic Seq2Seq decoding and teacher-vs-model-input choice, while introducing the idea from the decoder-input level.

What you can do after

Apply it to train sequence decoders that learn faster while still supporting autoregressive prediction later.

How it connects

Connects to Seq2Seq, autoregressive decoding, cross-entropy loss, and attention.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Graph Neural Networks in ActionTeacher forcing is a sequential-model training technique that supplies the ground-truth previous output as input when predicting the next step.

Compare how 2 books explain teacher forcing →

Related concepts

• recurrent neural networkuses · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• labeluses · A label is a value or annotation associated with an item that identifies its target category, outcome, or reference answer for learning or evaluation.
• tokenuses · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.