Regularization

A model that’s too simple does poorly even on the data it trained on: underfitting. A model flexible enough to fit its training data almost perfectly ends up fitting that data’s specific noise: overfitting, which is high variance. With a fixed amount of training data, more flexibility trades bias for variance and you cannot drive both down at once.

A network with large weights can carve an overly wiggly decision boundary that memorizes training noise instead of the underlying pattern. Regularization adds a second goal to training beyond “minimize loss”: also keep the weights small. L2 pulls every weight toward zero in proportion to its own size. L1 pulls every weight by the same fixed amount regardless of size, which is why small weights can reach exactly zero and stay there.

One convention worth knowing because every framework implements it and few explanations mention it: regularization is normally applied to weights only, not biases. A weight controls how strongly two things are connected, and shrinking it toward zero means “care less about this connection”, which is a meaningful thing to want. A bias just shifts a neuron’s threshold; shrinking it toward zero means “prefer a threshold of exactly zero”, which is not something you have any reason to prefer. Biases are also few in number, so penalising them buys almost nothing.

Formally, the expected squared error decomposes into three parts, where E[…] means “the average value you’d get if you repeated this over and over”:

Illustrative example, not from real data: imagine training each model twice on two different samples of the same data, and both times predicting the same input, whose true value is 0.7.

The flexible model is right on average and wrong every single time. It is never retrained into the version whose prediction is 0.7; it produces 0.2 or 1.2 depending on which noisy sample it happened to see, and you only ever get one of them. That is what “zero bias, high variance” means in practice, and it is why the model with the wrong average beats it here.

Note that the two decompositions agree with the direct calculation exactly, in both cases. That is not luck, it is the identity the formula states; the noise term is zero here only because this example has no measurement noise in it.

Here are both penalties inside a full update, with the data gradient present, so the two forces are visible at once. One weight w = 0.80 (positive, blue), one input x = 1.0, target y = 0.50. The lr = 1.0 and λ = 0.1 are a deliberately exaggerated pair, the same one the demo below uses, both far larger than anything you would train with, chosen so every step is legible in four decimal places.

The data gradient was +0.30 in all three rows, and it wanted the weight smaller anyway, because the prediction was too high. The penalty added to it. (The unregularized row lands on exactly 0.5000, the target, which is an artefact of lr = 1.0 meeting x = 1.0 and not a general property; at a realistic learning rate it would land short.) Had the data gradient been negative, wanting the weight larger, the penalty would have fought it, and the weight would have settled wherever the two forces balance. That balance point, not zero, is where a regularized weight ends up.

The single most-repeated claim about L1 is that it drives weights to exactly zero and L2 does not. Here is that happening, with the data gradient set to zero so the penalties are isolated, at the same lr = 1.0 and λ = 0.1, on three weights of very different sizes.

Those last two rows are sparsity, arrived at by arithmetic rather than by assertion. L1 takes the same 0.1 from every weight, which is 5% of a weight of 2.0 and more than the whole of a weight of 0.05, so small weights are the ones that run out, and they run out in order of size: 0.05 is gone after one step, 0.5 after five, and 2.0 would need twenty. Once a weight hits exactly 0 the clipping holds it there, and that connection is switched off for good. L2 takes the same percentage from every weight, so a weight approaching zero has less and less taken from it and never arrives.

The clipping is not an implementation detail; it is what makes L1 work. The w = 0.05 row is the case worth pointing at: one step would take it to −0.05, then the sign of the penalty would flip, and it would oscillate across zero forever without ever landing on it. Real implementations use a proximal step, which is the clip written formally: take the ordinary gradient step, then move the weight toward zero by lr·λ and stop if you reach zero.

Set w₀ to 0.50 and press ‘Take 5 decay steps’ twice. The blue L2 dot has moved from 0.5000 to 0.2952 to 0.1743 and its steps are visibly shrinking as it approaches the dashed zero line, which it will never touch. The orange L1 dot went 0.5000 → 0.4 → 0.3 → 0.2 → 0.1 → 0 in five equal strides and then stopped dead on the line, and its readout changed to “(locked at exactly 0)”. Equal strides against shrinking strides is the entire difference between the two penalties, drawn.

Back on the bias-variance demo in Figure 02: at degree 3 the solid curve tracks the dashed true curve and ignores individual dots. At degree 12 it passes through nearly every filled dot and misses the outlined ones by a mile - and the outlined dots are the only ones that matter, because they are the data the model never saw.

L2 adds (λ/2)‖w‖² to the loss, contributing a gradient of λw, so an SGD update becomes w ← (1−ηλ)w − η∇loss. The (1−ηλ) factor is literally where the name “weight decay” comes from. L1 adds λ‖w‖₁ to the loss, contributing a constant-magnitude gradient of λ·sign(w).

Those double bars are the two penalties written compactly: ‖w‖₁ is the sum of all the weights’ sizes ignoring sign, and ‖w‖² is the sum of their squares.

Elastic Net is both penalties at once, λ₁‖w‖₁ + (λ₂/2)‖w‖², and it exists because the two failure modes are different. L1 on its own, given several correlated inputs carrying the same information, tends to pick one of them arbitrarily and zero the rest, and which one it picks can change between training runs. Adding a little L2 makes it share the weight across the group instead. You get L1’s ability to switch connections off and L2’s stability about which ones.

And the link back to the top of this page, which is worth making explicit. Small weights produce a smoother decision boundary for a mechanical reason: a neuron’s output changes by w for every unit its input changes, so a large weight means a small change in the input produces a large change in the output, which is what a sharp, wiggly boundary is made of. Halve every weight in a layer and you halve how fast its output can swing. Regularization does not know anything about boundaries or overfitting; it only makes weights smaller, and smoother boundaries are what smaller weights look like when you plot them.

This connects directly to AdamW on the optimizers page: naively combining L2 regularization with an adaptive optimizer like Adam distorts the decay unevenly per parameter, which is exactly the problem AdamW’s decoupled weight decay fixes.