Vanishing Gradients

The gradient reaching an early layer is a product of every local slope between that layer and the output. Multiplying several numbers that are each less than 1 makes the result smaller every single time you add another one to the chain.

Sigmoid’s slope has a hard ceiling of 0.25, reached only at z = 0, and it falls away fast on either side: 0.1966 at z = 1, 0.1050 at z = 2, 0.0031 at z = 5.77. The optimistic bound over five layers is 0.25⁵ = 0.00098; the usually-quoted typical case is 0.2⁵ = 0.00032, which is what the demo shows at its defaults.

ReLU’s slope is exactly 1 whenever it’s active, so a chain of active ReLU units multiplies 1×1×1×1×1=1, no depth-driven shrinkage at all. But that’s not the whole story: a ReLU neuron that’s always inactive has slope exactly 0, and once that happens it stops learning permanently, a “dead ReLU.” ReLU trades gradual vanishing for a different failure mode, not a free pass.

Vanishing has a mirror image: exploding. The chain rule doesn’t just multiply activation derivatives together - at each layer it also multiplies in that layer’s outgoing weight. If the weights are large, the per-layer factor can climb above 1, and the exact same compounding multiplication that made the gradient vanish now makes it explode.

Before any powers of anything, here is a real gradient shrinking in a real network, computed the same way as the backpropagation walkthrough. The network is deliberately as simple as it can be while still being deep: five layers, one sigmoid neuron each, every weight set to 0.6 (positive, so blue), every bias 0, input 1.0.

Now backward. Start with a gradient of 1.0 arriving at the output, and pass it through each layer. The gradient travels from the last layer to the first, so this block and the lists below it run layer 5 first, the reverse of the forward table. Each pass multiplies by that layer’s slope, then by the weight:

Five layers, and 99.994% of the signal is gone. Nothing unusual was done to produce that; the weights are a perfectly ordinary 0.6 and the network is only five layers deep.

Look at what the per-layer factor is actually made of, because this is where most explanations quietly cut a corner. It is 0.145, not 0.24. Sigmoid’s slope contributes about 0.24, and the weight contributes 0.6, and it is their product that gets multiplied five times.

That also explains why the demo on this page reports a different number. It runs at a weight scale of 1.0 and a pre-activation of 1.0, so its per-layer factor is sigmoid’s slope on its own, sigmoid'(1.0) = 0.1966. This trace uses a weight of 0.6, so its factor carries that 0.6 as well and comes out at 0.145. Neither is wrong; they are two settings of the same knob, and the demo’s weight-scale slider is exactly the knob that moves between them.

“The training signal has all but disappeared” is worth converting into something you can picture. Here is what each layer’s weight actually receives from that single backward pass, at lr = 0.1:

That number, 1,401×, is the practical meaning of vanishing gradients. The network is not broken and it is not frozen; the last layer is learning at a normal speed while the first layer is effectively standing still. And the first layer is the one deciding what raw features everything else is built on. You get a network that trains, produces a falling loss, and has a bottom half that is still essentially the random numbers it was initialised with.

The fix has to change the per-layer factor, so it is worth seeing what each activation function contributes to it. Same five-layer network, same w = 0.6, same input, and the factor lists again run layer 5 first:

tanh beats sigmoid because its maximum slope is 1.0 rather than 0.25, four times higher, and that factor of four compounds. ReLU’s slope is exactly 1 wherever it is active, so its per-layer factor is the weight and nothing else, and the last row is the case everyone quotes: with weights near 1, the gradient passes through untouched. This is the entire argument for the ReLU family, now with the numbers behind it, and the dead-ReLU caveat is the price.

And the mirror image, in numbers. A per-layer factor of just 3, which one bigger weight is all it takes to produce, compounding over 5 layers gives 3⁵ = 243. Over 10 layers it is 3¹⁰ = 59,049. The update to a weight becomes a wild, oversized jump instead of a careful nudge, and training can diverge outright - loss shooting up instead of down, or turning into NaN.

The axis in Figure 02 is log-scaled and centred on 1.0, so every gridline is a factor of ten and growth and shrinkage are mirror images rather than two unrelated pictures. At the defaults - sigmoid, depth 5, weight scale 1.0 - the per-layer factor reads 0.1966 and the final value 2.9e-4, which is the 0.2⁵ figure in rung 1 rather than the trace’s 0.0000610, because this widget runs at weight scale 1.0 and the trace runs at 0.6. Both are correct and rung 2 says why.

Now change one thing at a time. Depth 5 to 8: the last bar drops three more gridlines. Sigmoid to tanh: every bar rises, because tanh’s slope at the same point is 0.41997 rather than 0.19661, more than double. Tanh to ReLU: the bars go flat, all at exactly 1.0, because ReLU’s slope is exactly 1 and the factor is the weight scale and nothing else. Tick ‘Make some neurons dead’: the bars go to zero and stay there, and the readout says “blocked by a dead neuron” - not small, blocked.

The weight scale slider is what turns this from two separate demos into one: at 1×, it reproduces the vanishing behavior from the top of this page exactly. Push it below 1× and the bars shrink faster - smaller weights, faster vanishing. Push it above 1× and the bars flip to growing instead - the chart is centered on 1.0 specifically so growth and shrinkage read as mirror images of the same underlying multiplication, not two unrelated phenomena.

There is one expression on this page and everything else is a consequence of it. The gradient reaching layer k from the output is the product, over every layer in between, of that layer’s activation slope times that layer’s outgoing weight. Two ingredients per layer, multiplied, then multiplied again across layers.

The standard fix for the exploding side is blunt and effective: gradient clipping. Before applying an update, check the size (the norm) of the gradient vector; if it’s above some threshold, rescale the whole vector down to that threshold before taking the step. Direction is preserved, only the size is capped. It doesn’t fix whatever’s causing the explosion, but it stops one enormous update from wrecking days of otherwise good training, which in practice is usually enough.

Clipping is one line of arithmetic, so here it is performed. The “norm” of a gradient is the same length measurement from the gradient descent page: square every component, add, take the square root.

Every component was multiplied by the same 0.322078, which is why the direction is preserved exactly. That is the difference between clipping by norm and clipping each component separately at some value: the latter changes the direction of the update, the former only its length. Norm clipping is the one used in practice, and clip_grad_norm_ is the function you will see in real training code.

Clipping stops explosions and does nothing for vanishing. Four other things do, and all of them appear elsewhere on this site, which is worth collecting in one place because a reader finishing this page is entitled to know that the problem has been solved rather than just described.

Two of those have their own pages here: scaled initialisation and normalization layers. Residual connections are covered in the convolutional networks module.

The reason to know all five rather than just the first is that they attack the same multiplication from different sides, and deep networks in practice use several at once. A modern residual network uses He initialisation, ReLU-family activations, normalization layers and skip connections together, and each one is aimed at the same product of per-layer factors that this page has been computing.