Exploding gradient

As explained in Transformers in Action

The exploding gradient problem is a training failure in which gradients become excessively large, causing unstable parameter updates and potentially preventing convergence.

An exploding gradient is a gradient that becomes excessively large during backpropagation. It can produce drastic, unstable weight updates, erratic learning, and failure to converge. It is the opposite of a vanishing gradient, which becomes too small to support effective learning. section 2.1.4

In a recurrent neural network, gradients are multiplied across recurrent steps. Long or complex sequences can compound their magnitude, making it harder to learn long-term dependencies. Gradient clipping limits gradient magnitude during training to help stabilize updates. section 2.1.4

Two training situations

A sequence-to-sequence model trained on lengthy, repetitive text or numeric sequences with extreme values may develop large gradients as recurrent steps accumulate. In scaled dot-product attention, large query–key dot products present a different stability risk: dividing them by √dₖ before softmax controls their magnitude and helps avoid softmax saturation and unstable gradients. section 2.1.4 section 2.2.3

Listing 2.3 Simplified scaled dot-attention implementation

embed_1 = np.array([0, 1, 0]) #1
embed_2 = np.array([1, 0, 1])
embed_3 = np.array([0, 1, 1])
embed_4 = np.array([1, 1, 0])

embeddings = np.array([embed_1, embed_2, embed_3, embed_4]) #2

Wq = rand(3, 3)  #3
Wk = rand(3, 3) 
Wv = rand(3, 3)

Q = embeddings.dot(Wq)  #4
K = embeddings.dot(Wk)
V = embeddings.dot(Wv)

attention_scores = softmax(Q.dot(K.T) / sqrt(K.shape[1]), axis=1) #5

attention_output = attention_scores.dot(V) #6

The attention-score calculation divides Q·Kᵀ by the square root of the key-vector dimension before applying softmax; the listing uses assigned embeddings and simplified weight matrices rather than a training loop. section 2.2.3

Figure 2.13 This figure shows that the variance of the scaled dot-product values is smaller than the variance of the unscaled dot-product values, indicating that scaling helps to control the variance of the dot-product values. This, in turn, helps prevent the gradients from exploding during backpropagation, improving the stability and convergence of the training process.

The figure contrasts scaled and unscaled dot-product values to show how scaling reduces their variance and supports more stable training. section 2.2.3

How Transformers in Action covers it

How deep it goes

Explains recurrent gradient growth and attention-score scaling, but gives no clipping threshold or worked training diagnosis.

Examples

Uses long repetitive sequences, extreme-valued numeric sequences, a scaled-versus-unscaled variance figure, and a simplified attention listing.

What it assumes

Assumes familiarity with gradients, backpropagation, and model weights while defining gradient clipping and explaining attention scaling.

What you can do after

Diagnose/debug with it by recognizing unstable gradient-driven updates and identifying clipping or attention-score scaling as relevant mitigations.

How it connects

Connects exploding gradients to vanishing gradients, backpropagation, recurrent neural networks, gradient clipping, softmax, and scaled dot-product attention.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionAn exploding gradient is a training condition in which propagated derivatives grow excessively large and destabilize optimization.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA training instability where gradient magnitude becomes very large, leading to excessively large parameter updates.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksAn exploding gradient is a neural-network training failure in which backpropagated gradient values become extremely large across successive layers.
• 

  liveBookmanning.com
  Machine Learning System DesignAn exploding gradient is a training condition in which gradient magnitudes become excessively large and destabilize optimization.

Compare how 5 books explain exploding gradient →

Related concepts

• gradient clippingaddressed by · Gradient clipping is an optimization technique that limits gradient values or their norm before parameter updates to reduce excessively large or unstable updates.
• normalizationaddressed by · Normalization is the process of transforming a value, representation, or data structure to satisfy a specified standard, scale, or structural constraint.
• weight initializationaddressed by · Weight initialization is the process of assigning starting values to a model’s trainable parameters before training or optimization.
• batch normalizationaddressed by · Batch normalization is a neural-network technique that normalizes activations using statistics computed over a mini-batch, with learned scale and shift parameters and typically running statistics for inference.
• recurrent neural networkproduced by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.