Gradients and Gradient Descent

The loss is a landscape. A weight is a ball on it. The height of the ground is how wrong the network currently is, and the slope of the ground directly under the ball is the gradient. Step downhill, repeatedly, and the network gets better.

With more than one parameter there is no single slope, so there is a list of them, one per parameter. Each entry is a partial derivative: nudge that one parameter, hold every other one perfectly still, and write down how much the loss moved. Do that once per parameter and the list you end up with is the gradient.

Read as a direction rather than as a list, the gradient points in the direction of steepest increase of the loss. You want the loss to go down. The gradient points up. So you step against it, and that is the whole of the minus sign.

Take the simplest possible loss landscape, loss(x) = x². Its slope at any point turns out to be 2x, but rather than trusting a formula out of nowhere, check it the honest, no-calculus way: nudge x by a tiny amount and see how much the output moves.

That’s the honest, no-calculus definition of a derivative: how much the output changes for a tiny change in input, divided by that tiny change. Notice the answer came out to 6.001, not exactly 6 - the smaller the nudge, the closer this gets to the true slope; shrinking the nudge all the way toward zero is what calculus calls the derivative. This is the same trick already used to verify the derivative table on the activation-functions page, and it’ll come back once more on the output-activations page, so it’s worth having solid here.

One gradient descent step is just x_new = x − lr·gradient(x). Worked example: starting at x=2.4 with lr=0.35, the gradient is 2×2.4=4.8, so the next value is 2.4 − 0.35×4.8 = 0.72, much closer to the bottom (x=0) already.

Now the gradient of an actual loss with respect to two actual parameters, computed by hand. This is the same kind of network the loss functions page left off with, now taken the rest of the way: ŷ = w·x + b with w = 0.50 (positive, so blue on a diagram) and b = 0.00, on two training points. The loss is the average half-squared error, this site’s convention, which is why every gradient below comes out without a stray factor of 2 in it.

Three things to take from that. Both entries of the gradient were negative, so both parameters went up, and they went up by different amounts, because w had the steeper slope. Nobody decided that; it fell out of the arithmetic.

The “length” line is worth reading twice, because it is the first place on this site where two numbers get combined into one. It is the ordinary distance formula: square each entry, add, take the square root. That single number is how big the whole gradient is, ignoring which direction it points, and it is shrinking, which is what “approaching a minimum” looks like numerically: the ground under the ball is getting flatter. At an exact minimum, every entry of the gradient is 0, the length is 0, and the updates stop by themselves. This same quantity comes back on the vanishing gradients page under its other name, the norm, where clipping is defined in terms of it.

And look at how dL/dw got built: (−0.500000×1.0 + −2.000000×2.0) / 2. The second example contributed −4.0 of the −4.5 total, 89% of the whole signal, because its error was four times as large and its input was twice as large. Each example’s vote is weighted by its own error and its own input. Nothing about that is specific to this loss; it is what “the gradient of an average” means, and it is why one badly-fit example with a large input can dominate a whole batch.

Below about lr=0.5, the ball glides smoothly to the bottom. At exactly lr=0.5 it reaches the bottom in a single step. Between 0.5 and 1 it overshoots past the bottom and bounces to the other side, but each bounce is smaller than the last. At exactly lr=1, it bounces forever at the same distance, never settling and never escaping. Above lr=1, it overshoots farther every single step. Those claims are checkable, so here they are checked. Starting from x=2.4 on loss(x) = x², one step is x ← x − lr·2x = x·(1 − 2·lr). The whole behaviour is decided by that one multiplier.

The rule behind all six rows: the step converges when |1 − 2·lr| < 1, which for this landscape means lr < 1. The bars in |…| mean “ignore the sign”, so a multiplier of −0.80 counts as 0.80 and shrinks the number it multiplies, while −1.20 counts as 1.20 and grows it. The 2 in that expression is how sharply x² bends, its curvature: the slope of x² is 2x, which gains 2 for every 1 you move sideways, and a bowl whose slope changes faster means a smaller maximum safe learning rate. That is not a quirk of this toy function, it is the constraint that decides learning rates in real training, and it returns with real numbers on the optimizers page.

Every row of that table is a setting of the slider in Figure 02, and the fading trail is the row drawn. At lr = 0.10 the trail is a line of dots marching one way. At lr = 0.90 the dots alternate left and right of the bottom, each one closer in. At lr = 1.00 there are only ever two dots, because the ball visits exactly two places forever. Above 1.00 the trail runs off the edge of the frame and the ball sticks against it - at that point stop reading the picture and read the weight number underneath, which keeps going.

One gradient descent step is x_new = x − lr·gradient(x), where lr controls how big a step to take.

The claim that the gradient points in the direction of steepest increase is the one the minus sign rests on, so here it is checked rather than asserted. Stand at the same starting point, w = 0.500000, b = 0.000000, where the loss is 1.062500. Move exactly the same distance, 0.1, in five different directions and see what the loss does. Each direction is written as “how much of the step goes into w, how much into b”, scaled so the total distance moved is 0.1 every time.

Every direction with a downhill component lowers the loss, and the one built from the gradient lowers it the most: 0.240292 against 0.231237 for the nearest competitor, and more than twice what moving b alone achieves. Turn the direction around and it becomes the single fastest way to make the loss worse, which is the same fact stated from the other side. That is what “steepest” means, and it is why the update subtracts the gradient instead of adding it.

Nothing about the five-direction check depends on the loss being squared error or on there being two parameters; with a million parameters the arithmetic is the same and only the list is longer.

Two honest limitations, because the picture above is deliberately friendly.

x² has exactly one minimum and gradient descent cannot fail to find it. A real loss landscape has many. It can have local minima, dips that are not the deepest dip available, and far more commonly it has saddle points, places that slope down in some directions and up in others, where the gradient is near zero in every direction and progress stalls without the loss being anywhere near as low as it could get. In very high-dimensional networks saddle points are the more serious of the two, simply because a point needs to curve upward in every one of a million directions at once to be a true local minimum, and that is rare. Most of what optimizers do beyond plain gradient descent is about getting through flat and awkward regions faster.

And nothing here says when to stop. Gradient descent has no built-in finish line; the gradient shrinks toward zero but never quite arrives. In practice you stop after a fixed number of steps, or when the loss stops improving, or when a separate held-out score stops improving, which is early stopping.