Loss Functions

Squared error is the default loss for regression, meaning any task where the network has to produce a number rather than pick a category: a house price, a temperature, a position. You take the gap between the prediction and the right answer, and you square it.

Squaring does two things at once, and both are on purpose. It makes the score positive, so being 2 too high and 2 too low both cost the same, and it makes the cost grow faster than the mistake does. Being twice as wrong is four times as expensive. That second property is what makes squared error push hard on the worst predictions first.

Squared error is the wrong tool the moment the network’s job changes from “produce a number” to “pick a category”. For classification the network outputs a probability, a number between 0 and 1 saying how confident it is, and what you want to punish is not distance, it is misplaced confidence. Being 90% sure of the wrong answer should hurt much more than being 55% sure of the wrong answer, and it should hurt more than twice as much.

Cross-entropy does exactly that, and the mechanism is one idea: score the probability the network assigned to the answer that actually happened, and take the negative natural log of it. That is all −ln(ŷ) means. The log is there because it has one useful shape: it is 0 when the probability is 1, small when the probability is high, and it grows without limit as the probability approaches 0. There is no ceiling on the punishment for confidently ruling out the truth.

Here is a full trace, batch and all. The network is as small as it can be while still having something to learn: one input x, one weight w = 0.60, one bias b = 0.10, and no activation function on the output, so the prediction is just ŷ = w·x + b. The weight is positive, so it would be drawn blue on any diagram on this site. Three training examples go through it.

All three errors are negative, so the network is guessing low on every example. Look at which example the loss is mostly made of. The third one contributed 0.3600 of the 0.5400 total, two thirds of the whole score, off an error only twice the size of the other two. That is the squaring at work, and it is the reason this loss will end up caring about that example more than about the other two combined.

What happens next is that this one number, 0.0900, gets turned into a change to w and a change to b. That is not a property of squared error, it is the same procedure for every loss on this page, and it is the subject of the next page, which picks up this exact kind of network and walks the arithmetic.

That p = 0.02 row is the factor of eight from the top of this page. Squared error scores the same prediction at 0.4802; cross-entropy scores it at 3.9120, 8.15 times higher, for being confidently wrong about the same example.

Here is the full path from a raw network score to a loss, which is the part that usually gets skipped. The network’s last layer produces a raw, unbounded score called a logit, written z. Sigmoid squashes that logit into a probability. Cross-entropy scores the probability.

The third example is the interesting one. The network said 0.574, barely better than a coin flip, and got it “right” in the sense that 0.574 rounds to 1. Accuracy would score that as a win. Cross-entropy scores it as the worst of the three, worse than the example the network got wrong less confidently. Cross-entropy grades confidence, not correctness.

Now the outlier comparison. To see how far apart squared error and MAE are, compare each average against what the three well-behaved errors would have produced on their own. Three errors of 0.1 give a squared-error average of 0.01 and an MAE average of 0.1. Adding the single outlier takes those to 6.2575 and 1.325 respectively.

The two averages are not directly comparable to each other, one is in squared units and one is not, which is exactly why the comparison has to be made against each loss’s own baseline. Measured that way, one outlier out of four moves squared error roughly 47 times more than it moves MAE.

Mean absolute error itself is the plain distance between prediction and answer, |ŷ−y|. Take the same wrong prediction the demo above used, ŷ=0.02 and y=1:

That squared-error figure uses this page’s ½(ŷ−y)² convention. If you check the arithmetic yourself with the plain (ŷ−y)² form instead, you’ll land on 0.9604 for this same example, not a mistake, just a different convention.

Huber loss behaves like squared error for small mistakes and like MAE for large ones. The switch happens at a threshold you choose, δ. Worked with δ=1.0:

Huber caps how much a single outlier can dominate training, while still behaving exactly like squared error close to the target. It’s the loss behind the original DQN reinforcement-learning agent, among other places you’ll run into it.

Hinge loss, for a label y ∈ {−1, +1} and a raw score f(x), asks a coarser question than any of the above: is your raw score on the correct side of the boundary, by a comfortable margin?

Set the prediction to 0.50 with the answer at +1 in Figure 02 and read the three losses off the switch: squared error 0.1250, MAE 0.5000, Huber 0.1250. Huber and squared error agree exactly, because an error of 0.5 is inside Huber’s δ=1 threshold and Huber is squared error in there. Now drag the prediction to −1.00, an error of 2.0: squared error 2.0000, MAE 2.0000, Huber 1.5000. The two that agreed have separated, and the two that disagreed have crossed.

The red bar on the number line is the same length in every one of those readings - the bar is the error, the number underneath is what each loss thinks that error is worth.

The widget’s own static cross-entropy panel is the fourth reading the number line cannot draw: 0.480 against 3.912 for the identical prediction of 0.02. There is no bar length that shows a difference of eight times, which is exactly why that panel is separate.

The full binary form of cross-entropy, L = −[y·ln(ŷ) + (1−y)·ln(1−ŷ)], looks like two terms but only ever uses one of them. When the true label y is 1, (1−y) is 0 and the second term vanishes, leaving −ln(ŷ). When y is 0, the first term vanishes, leaving −ln(1−ŷ). It is an if-statement written as arithmetic, which matters because the training procedure needs a loss that is differentiable, meaning it has a well-defined slope everywhere rather than jumping between two branches.

MAE’s derivative is dL/dŷ = sign(ŷ−y): always exactly +1 or −1, no matter how far off the prediction is. Every mistake, big or small, pushes the weights by the same amount. Squared error’s derivative grows with the size of the mistake instead, so a huge outlier produces a huge gradient and one bad example can dominate an entire weight update.

With more than two categories the network produces one logit per class, softmax turns the whole set of logits into probabilities that sum to 1, and cross-entropy scores the one belonging to the correct class. Nothing about the idea changes; there are just more numbers.

Two pieces of vocabulary arrive with them. A distribution here means a list of probabilities covering every possible outcome and adding up to exactly 1, so [0.659, 0.242, 0.099] is one. And one-hot is how the correct answer gets written in the same shape: one number per class, a 1 in the position of the right answer and 0 everywhere else. “The answer is class 1, out of three” becomes [1, 0, 0].

If the true class had been class 3 instead, the same forward pass would have scored −ln(0.098566) = 2.317030 instead of 0.417030, five and a half times the loss, from identical logits. The loss depends on which answer was correct, not on how spread out the predictions were. Softmax gets its own page, Softmax for Multi-Class Output.

Now the reason cross-entropy is the standard partner for a sigmoid or softmax output, and squared error is not. The question to ask is: when the raw logit z moves a little, how much does the loss move? That quantity is written dL/dz, and for now it can be read exactly as those words - how much the loss changes when z changes - with no more machinery than that. Where such numbers come from, and what training does with them, are the next page’s subject. All that matters here is that a loss which barely moves when the network is badly wrong is a loss that will barely teach it anything.

At z = −4 the network is confidently, badly wrong: it says 1.8% for something that is true. Cross-entropy responds with −0.982, almost the largest response it can give. Squared error responds with −0.0173, 56.6 times smaller, because it multiplies the error by sigmoid’s own slope, and sigmoid’s slope is nearly zero out there. Push the logit out to −8 and the gap widens to over 2,900 times; the further into the wrong answer the network goes, the quieter squared error gets. Squared error plus sigmoid produces almost no learning signal precisely when the network is most wrong. Cross-entropy’s p − y cancels that slope term away exactly. That cancellation is the whole reason the pairing exists, and it is derived properly on the output activations page.

A precision note: p − y is the response with respect to the logit, after sigmoid has been folded in. The response with respect to the probability itself, dL/dp, is (p − y) / (p(1−p)), which does blow up near 0 and 1. Frameworks combine the sigmoid and the cross-entropy into one operation (BCEWithLogitsLoss in PyTorch) so that the clean p − y form is what actually gets computed and the unstable intermediate never appears.

The last one is not really a loss you would pick off a menu, it is a way of understanding the one you already picked. KL-divergence answers “how much worse is my predicted distribution than the true one?”, using “distribution” in the sense defined above: a list of probabilities over every outcome, adding to 1. Cross-entropy answers “how expensive is my predicted distribution?”, and part of that expense is unavoidable: even a perfect prediction costs something, because the true distribution has its own built-in uncertainty. KL-divergence is cross-entropy with that unavoidable part subtracted off, so a perfect prediction scores exactly 0.

Cross-entropy decomposes as H(P,Q) = H(P) + D_KL(P‖Q), so minimizing cross-entropy and minimizing KL-divergence are the same optimization problem in practice - H(P), the true data’s own uncertainty, doesn’t depend on the model at all. Quick worked check: true distribution P=[0.7, 0.3], predicted Q=[0.5, 0.5] gives H(P)=0.610864 and H(P,Q)=−ln(0.5)=0.693147, so D_KL=0.082283, matching a direct KL calculation, 0.7×ln(1.4) + 0.3×ln(0.6) = 0.082283. You’ll see this term again if you go on to study generative models.

Six losses is a lot to hold at once. The choice is usually made by the shape of the output, not by taste:

If you are unsure, the shape of your output layer has already decided for you. That is not a coincidence; the page on output activations is the other half of this one.