Training-Time Regularization

The previous page covered regularizers that change the loss formula itself. These four are applied to the training procedure or the network’s structure instead.

Dropout switches off a fraction of neurons at random on every forward pass, as if training a huge ensemble of thinner sub-networks that all share weights. No single neuron can be relied on, so the network spreads information across redundant paths. At test time the whole network is used, so survivors are scaled up during training to keep the expected total signal the same.

The p trap: this page uses p = probability of being dropped, which is PyTorch’s convention. CS231n and others use p = probability of being kept, with 1/p scaling rather than 1/(1−p). Identical behaviour, opposite letters.

Batch normalization forces every layer’s inputs to a fixed mean of 0 and spread of 1 within each mini-batch, then hands back a scale and a shift so the network can undo it where it needs to.

Early stopping needs a validation set: training data to learn from, validation data to make decisions with, test data scored once at the end and never used to decide anything. A typical split is 80/10/10. Training loss cannot tell you when to stop because it falls essentially forever; validation loss can, because the network never adjusted itself to fit it.

One mechanism used below and worth naming now: a running average blends each new measurement into a stored value instead of replacing it - keep 90% of what you had, add 10% of what you just saw. You do it this way because any single mini-batch’s mean and variance are noisy, and inference needs one fixed pair of numbers rather than whichever batch happened to arrive last.

Dropout. p=0.5, layer activations h=[0.8, −0.3, 1.2, 0.5], and this particular random draw happens to keep units 1 and 3:

One random draw does not show an ensemble. Here are two draws over the same layer, plus the average over every possible draw:

Unbiased on average is not the same as unchanged. Each individual forward pass is noisy, and the scaling makes that noise larger, not smaller. For a unit with activation h, the variance introduced is h²·p/(1−p):

That noise is the regularizer. It is also why p = 0.8 is rarely a good idea: past a point, the layer’s output is more noise than signal. Common values are 0.5 for fully connected hidden layers and 0.1 to 0.2 for convolutional ones, which have far fewer parameters per unit and need less help.

The backward pass needs no special rule, which is worth seeing rather than being told. The mask is just a multiplication by a constant, so the gradient gets multiplied by the same constant on the way back:

A unit that was dropped receives exactly zero gradient, so its incoming weights are not updated on that pass at all. That is the second half of dropout’s regularizing effect: not only does the network have to work without any given unit, no unit gets to learn from every example.

That is dropout in full. Next, a technique that regularizes not by changing what a layer computes but by changing when training stops.

Early stopping. Track validation loss per epoch, stop when it hasn’t improved for some number of consecutive epochs (“patience”), and keep the weights from the BEST epoch, not the last one.

Train loss falls steadily through all 10 epochs. Validation loss bottoms out at epoch 6 (0.30), then climbs for the rest of training. With patience = 3, here is exactly when the stop fires and which weights survive:

Two things that trip people up. Patience exists because validation loss is noisy: a single bad epoch is not evidence of overfitting, and a patience of 1 would stop most runs far too early. And the epochs after the best one are not wasted; you had to run them to find out that epoch 6 was the best. Early stopping always costs you patience epochs of hindsight.

That is the second technique in full. The third, batch normalization, is the longest block here - it has more moving parts, so take it in stages: first what it computes, then why the tiny constant in its formula matters, then how its two learned knobs actually get updated.

Batch normalization. Mini-batch of 4 pre-activation values x=[2.0, 4.0, 4.0, 6.0]:

Before the next block, what ε is doing in that denominator. It is a tiny constant, usually 1e-5, added to the variance purely so the division cannot blow up. If every value in a mini-batch happens to be identical, the variance is exactly 0, and without ε you would divide by zero. With ε you divide by √0.00001 = 0.003162 instead and every normalized value comes out as 0, which is the sensible answer: values that were all identical carry no information to spread out.

ε is invisible in normal operation and decisive in the one case that would otherwise crash. That is what a numerical-stability constant is for. The demo in rung 3 uses the same ε = 1e-5, so you can watch it happen: drag the four points close together and the normalized values stop growing the way the top row of that block does.

Three words in the next block mean something different here from what they meant earlier in this module. “Momentum” here is not the Momentum optimizer. On the optimizers page, momentum is accumulated velocity and its coefficient β = 0.9 says how much of the past you keep. Here, “momentum” is the weight given to the new value when updating a running average, so a momentum of 0.1 means the stored value keeps 0.9 of itself and takes only 0.1 of the incoming batch. The two numbers look like complements of each other and the roles are reversed: 0.9 there means long memory, and 0.1 here also means long memory. Second, β in y = γx̂ + β is batch norm’s shift parameter, unrelated to the optimizer’s momentum coefficient and to Adam’s β₁ and β₂. Third, “biased” below is a property of a formula: the version that divides by m systematically under-estimates the true spread of the population the sample came from, and dividing by m−1 instead corrects it - a third unrelated sense of a word this module has now used three ways.

Two details in there are easy to miss and both are real. The variance used to normalize the batch is the biased one, dividing by m; the variance stored into the running average is the unbiased one, dividing by m−1, which for a batch of 4 is 4/3 times larger. PyTorch does exactly this and documents it. And the running statistics are a slow-moving average, so at momentum 0.1 they take dozens of batches to forget an old value, which is why a network switched to evaluation mode too early in training can produce noticeably worse predictions than it does in training mode.

Pause and recap before the last piece: so far this block has covered what batch norm computes on a forward pass, and why ε is there. What is left is the one thing that makes γ and β genuine trainable parameters rather than fixed constants - seeing them actually move in response to a gradient, worked by hand once below.

Calling γ and β “learnable” is only meaningful if you can see them learn. They are ordinary parameters with ordinary gradients. The block below runs one step of the machinery that the next page is entirely about, one page early. You need one idea: a “gradient arriving at y” is just the answer to if this output number changed by a little, how much would the loss change?, one such answer per output value.

γ went up, meaning the network decided this layer wanted slightly more spread than the forced spread of 1. That is the escape hatch working: normalization is imposed, then the network is handed two knobs and allowed to undo as much of it as it turns out to need. If the optimal γ for a layer were √(σ²_B+ε) and the optimal β were μ_B, the layer would learn to cancel itself out entirely.

One practical warning, since this page covers both techniques. Dropout and batch normalization directly interfere with each other, and the reason is in the numbers above: dropout deliberately adds variance to a layer’s output, and batch norm’s running statistics are estimated during training, when dropout is active, but used at inference, when it is not. The statistics therefore describe a distribution the network never actually sees at test time. The usual practice is to use one or the other in a given block, or to place dropout only after the last normalized layer. If a network is unexpectedly worse in evaluation mode than in training mode, this is the first thing to check.

On the dropout demo in Figure 02: press ‘Resample mask’ repeatedly at p = 0.5 and watch the four hidden values. Every draw is a different sub-network, and the greyed dashed circles are the units contributing nothing on this pass - not a small contribution, exactly zero, in both directions. Now switch to Inference: all four are live, at their original values with no scaling at all, and the resample button is disabled because there is nothing left to randomise.

On the batch-norm demo below: drag the four points until they are almost on top of each other, around [3.0, 3.0, 3.0, 3.1]. σ²_B collapses to 0.0019 and the divisor to 0.0434, and x̂ still only reaches +1.7275 rather than running away, because ε is holding the floor under the division. Drag them back apart and ε becomes invisible again - on the default batch it changes the divisor by 0.0000035. That is what a numerical-stability constant is: nothing at all, until the one case where it is everything.

Every other technique on this page constrains the model. Data augmentation attacks the problem from the other end: overfitting happens when a model has more capacity than it has data, and if you cannot reduce the capacity, you can manufacture more data. The rule is that the transformation must be label-preserving. A photo of a cat rotated 15 degrees is still a photo of a cat, so the network can be shown it with the same label and learns that “cat” does not depend on exact orientation.

The label-preserving requirement is stricter than it first sounds, and it is where augmentation goes wrong. Flipping a photo of a cat horizontally is fine. Flipping a photo of the digit 2 horizontally produces something that is not a 2, and training on it as one actively teaches the network something false. Rotating a 6 by 180 degrees produces a 9. Whether a transformation is safe depends entirely on the task, and there is no general answer, only a question you have to ask about each one.

Two mechanical notes. Augmentation is applied to the training set only, never to validation or test, because those exist to measure performance on real data. And it is normally applied on the fly, freshly randomised each epoch, rather than generated once and saved, so the network sees a different variant of each example every time it comes around, for the same reason dropout resamples its mask every forward pass.

It’s the same idea as dropout’s noise-based regularization, just injected into the inputs instead of the activations.