Optimizers

Every optimizer takes the same input, a gradient, and produces the same kind of output, a change to the weights. What differs is only how much of the past each one remembers and what it does with that memory.

Momentum. Plain gradient descent’s “ball” has no memory - every step is decided purely by the current slope, so on a landscape shaped like a narrow valley it can zig-zag across the valley walls instead of smoothly rolling toward the far end. Momentum gives the ball velocity that accumulates: it remembers recent downhill directions and keeps drifting that way even where the current slope briefly disagrees, canceling the zig-zag and speeding up consistent progress.

Nesterov. Momentum computes the gradient where it currently stands, then adds the accumulated velocity and moves. But the velocity is going to carry it somewhere regardless, so the gradient it just measured is already slightly out of date by the time the move happens. Nesterov momentum measures the gradient at the place the velocity is about to take it, and uses that instead. It is the difference between braking when you feel the wall and braking when you see it.

AdaGrad. Momentum fixes direction. AdaGrad instead fixes the fact that one single global learning rate is wrong for every parameter simultaneously - some parameters (rarely-updated ones) need larger steps, others need smaller ones. AdaGrad tracks how much each parameter has moved historically and shrinks its personal step size the more it’s already moved. The known flaw: that running sum only ever grows, so the step size keeps shrinking and eventually grinds to a near-halt even if the model still has more to learn.

RMSProp. AdaGrad’s own fix for AdaGrad: keep the idea of a personal step size per parameter, but replace the ever-growing sum with a decaying running average, so old gradient history fades out and the step size can recover instead of only ever shrinking.

Adam. Combines momentum’s idea (track a running average of the gradient itself, for direction) with RMSProp’s idea (track a running average of the squared gradient, for a safe adaptive step size), plus one correction neither of those needs on its own: since both running averages start at zero, they’re artificially biased toward zero for the first few steps, so Adam explicitly corrects for that bias.

AdamW. One more refinement: naively adding L2 weight decay to Adam, by folding it into the gradient before Adam’s adaptive rescaling, means that rescaling ends up distorting the decay too, applying it unevenly per parameter. AdamW’s fix is to apply weight decay directly to the weights, completely separate from the adaptive gradient step.

Plain gradient descent wins this race, and it is worth saying plainly why rather than hiding it. Its step size is proportional to the gradient, and at x=2.4 the gradient is 4.8, so its first step is a large 0.48. Adam deliberately normalises its step to roughly ±lr regardless of gradient size, so its first step is exactly 0.1. On a single smooth bowl where the gradient is helpfully large, throwing that information away is a loss. Rung 5 shows the landscape where throwing it away is a win, with numbers.

Momentum and Nesterov overshoot past the bottom entirely by step 4, which is not a bug; that is what accumulated velocity does, and it is the price of the speed that gets them there in three steps instead of five.

Six blocks follow, each deriving one row of the table above by hand. They build on each other in pairs - momentum then Nesterov (both remember direction), AdaGrad then RMSProp (both scale the step per parameter), then Adam and AdamW (which combine the two ideas). Worth reading once in order; after that, each block stands alone as a reference.

Momentum, derived. Same x² landscape and starting point, x₀=2.4, lr=0.1, β=0.9:

Nesterov, derived in the same convention, so the two are directly comparable. The only change from plain momentum is where the gradient is evaluated: at the peeked-ahead position rather than the current one.

Step 4 is where the lookahead earns its keep. Plain momentum’s rule, applied standing where Nesterov now is at x=0.427008, would measure a gradient of +0.854016 and keep accelerating downhill. Nesterov peeks to x=−0.260045, discovers the gradient has already flipped to −0.520090, and starts braking a full step earlier. (Plain momentum, running its own race from the start, is at 0.148800 after three steps, which is why that number and not 0.427008 appears in its row of the comparison table; the comparison here is between the two rules evaluated at the same place.) The result is visible in the table: after 5 steps momentum has overshot to −1.393008 while Nesterov is at −0.623660, less than half as far past the bottom.

Momentum and Nesterov both still take a step whose size depends only on the learning rate and the accumulated velocity - every parameter is treated the same way. AdaGrad tries a different idea entirely: give each parameter its own step size, shrinking it for parameters whose gradient has historically been large.

AdaGrad, derived.

AdaGrad’s per-parameter shrinking never lets up, even once a parameter’s gradients calm down, since it keeps adding every squared gradient it has ever seen. RMSProp keeps the same per-parameter idea but forgets old gradients over time instead of piling them up forever.

RMSProp, derived.

Unlike AdaGrad, this doesn’t shrink forever - since s is a decaying average, it settles at a fixed point under a constant gradient, the exact fix RMSProp promised over AdaGrad’s permanent shrink.

Two ideas have now been built separately: momentum’s running direction, and AdaGrad/RMSProp’s per-parameter step-scaling. Adam is what happens when a single optimizer tracks both at once, plus one extra fix for the first few steps, when both running averages start at zero and are biased low.

Adam, derived, x₀=2.4, lr=0.1, same landscape:

This isn’t a coincidence: at t=1, the bias correction perfectly cancels the running-average weighting for ANY gradient value, so Adam’s very first step always moves by exactly ±lr, regardless of how large the raw gradient was.

AdamW, derived. The claim that folding L2 decay into Adam’s gradient lets Adam’s rescaling distort the decay is checkable. Adam divides every update by √ŝ, and √ŝ tracks the size of that parameter’s own recent gradients. So a decay term that got folded into the gradient gets divided by the parameter’s gradient size too. Take two weights in the same network, both at w = 0.5, both with λ = 0.1 and lr = 0.1. Weight A sits in a busy part of the network with a steady gradient of 2.0. Weight B is in a quiet part, gradient 0.002.

One λ was requested and, under Adam with L2, thirty-nine different strengths of decay were delivered, purely as a side effect of which weights happen to have large gradients. AdamW’s decay term sits outside the m̂/√ŝ fraction entirely, so it does exactly what it says regardless of gradient size.

Two things flagged as exaggerated or simplified. λ = 0.1 is far larger than anything you would train with; realistic values run from about 0.01 down to 0.0001, and 0.1 is used here only so the distortion is legible at eight decimal places instead of buried at twelve. And √ŝ ≈ |g| is an approximation that holds when a parameter’s gradient is roughly steady. During real training gradients move around and the exact ratio moves with them, but the direction of the distortion - big-gradient parameters get less decay than you asked for and small-gradient parameters get more - is not an approximation.

And the batch trade-off, in numbers. Four data points, targets=[1.8, 2.2, 1.9, 6.0], current weight w=2, using the gradient formula 2×(w−target) per point. Individual per-point gradients: [0.4, −0.4, 0.2, −8.0]. The full-batch average is −1.95. A single-example SGD step might draw gradient +0.4 or −8.0 depending on which point gets picked, wildly different directions from the exact same starting weight. A mini-batch of 2 (say, points 3 and 4) averages to −3.9, closer to the true full-batch gradient than any single point alone, though still not exact. As batch size grows toward the full dataset, the estimate’s noise shrinks toward the true gradient.

The three balls in Figure 02 are rows 1, 2 and 6 of that table, drawn. Press ‘Take a step’ three times with all three shown. Plain GD’s dots are evenly spaced and shrinking - each one is 80% of the last distance from the bottom. Momentum’s dots get further apart each time, which is velocity accumulating, and by step 4 it has crossed the bottom and its dots are on the other side. Adam’s dots are almost equally spaced and tiny, because Adam’s step is pinned near ±lr no matter how steep the ground is: at step 1 the gradient is 4.8 and Adam still moves exactly 0.1.

The fading trail is five steps long, so once a ball has settled the trail collapses into a single dot, and once a ball has diverged the trail runs off the frame and the ball sticks to it - at which point the numbers underneath are the only honest reading.

Two words first, because AdamW is built on them and they properly belong to a later page. Weight decay is a second pull applied to every weight on every step, always toward zero, on top of whatever the gradient asked for. It exists because large weights are what an overfitted network is made of, so shrinking them a little on every step is a cheap way to keep the network from memorising its training data. Its strength is a single number, λ (lambda): at λ = 0 there is no pull at all, and the larger λ gets, the harder every weight is dragged toward zero.

That is everything you need to read the AdamW line below. Why you would want it, what it does to a network’s behaviour, and the difference between the two ways of writing it down are the regularization page’s subject.

Every line above is written in one convention: v accumulates raw gradients and the learning rate is applied outside it. Papers vary, and the same letter v in another source may already have the learning rate and the minus sign folded into it. If two sources’ numbers will not reconcile, check that before checking the arithmetic.

The decay term in AdamW, on the right, is untouched by anything Adam is doing to the gradient. This is why “Adam with weight_decay=0.01” in most modern frameworks actually means AdamW under the hood.

The comparison table near the top of this ladder shows plain gradient descent beating every adaptive method. That result is real, and it is a property of the landscape, not of the optimizers. A single symmetric bowl is the best possible case for a single global learning rate, because one step size genuinely suits every direction.

Real loss landscapes are not symmetric. Change one thing about the test problem, make it two-dimensional with one direction twenty times steeper than the other, f(x, y) = ½(x² + 20y²), start at (5.0, 1.0), and the picture inverts.

That is the zig-zag the momentum section described in words, now written out as numbers. And it explains the real constraint, which is not “gradient descent is slow” but “gradient descent’s learning rate is set by its steepest direction and then every other direction has to live with that choice”. The multiplier for y stays inside (−1, 1) only while lr < 2/20 = 0.1. Push lr to 0.11 and y’s multiplier is −1.2 and the run diverges, even though the x direction would have been perfectly stable up to lr = 2. One direction is holding the whole run hostage, twenty times below the rate the other direction wants.

RMSProp and Adam divide each direction by the size of its own recent gradients, which decouples the two: y no longer sets x’s speed limit. On this small example that decoupling costs more than it saves over eight steps, because the shallow direction still only moves at lr per step. In a real network with millions of parameters and gradient magnitudes spanning several orders of magnitude, the decoupling is the whole ballgame, and it is why Adam is the default that most training runs start from.

Flagged as simplified: two parameters standing in for millions, and a quadratic bowl standing in for a loss surface that is not quadratic anywhere. The mechanism, per-direction step sizes versus one shared step size, is exactly the real one.

One more knob, separate from all of the above: every optimizer on this page still takes a learning rate lr as an input - none of them decide it for you. In practice, that learning rate is rarely held perfectly constant for the whole training run: a common pattern is a short warmup (starting small and ramping up, so the first few updates, before the running averages inside Adam/RMSProp have gathered enough history to be reliable, don’t take an oversized step) followed by decay (gradually shrinking the learning rate as training progresses, so later updates fine-tune instead of overshoot). That’s a schedule on top of whichever optimizer you pick - Adam with a decaying learning rate is still Adam, just with one more moving part layered on.