Softmax for Multi-Class Output

Sigmoid answers a yes/no question. "Cat, dog, or bird?" needs a probability distribution spread across several options at once, not an independent yes/no per option. Softmaxdoes exactly that: e to the power of one class's score, divided by the sum of e to the power of every class's score.

Before the formula, the requirements. You have three raw scores out of the last layer, one per class, and you want three probabilities. Whatever turns one into the other has to satisfy three things:

1. Every output is positive. There is no such thing as a -12% chance.
2. The outputs add up to exactly 1. If exactly one of the three classes is true, the confidence has to be fully distributed among them with nothing left over.
3. A bigger score gives a bigger probability. If the network scored "cat" above "dog", the cat probability must come out above the dog probability. The ordering must survive.

Why not just divide by the total?

The block at the top of this page, taken apart. Divide each score by the sum of all the scores. It costs nothing and it obviously sums to 1. On (2.0, 1.0, 0.1) it gives (0.645161, 0.322581, 0.032258), which looks fine. On (2.0, -1.0, 0.1) it gives (1.818182, -0.909091, 0.090909): still sums to 1, and complete nonsense, a probability of 181.8% and a probability of -90.9%. Requirement 1 is broken. And on (1.0, -1.0, 0.0) the sum is exactly 0 and the method divides by zero.

Both failures come from the same source: raw scores can be negative, and the fix has to be something that turns any number, however negative, into a positive one, without breaking the ordering. e^x does exactly that. It is positive for every input without exception (e^-1000 is a very small positive number, never zero and never negative), and it is strictly increasing, so a bigger score always produces a bigger e^score. Requirements 1 and 3 are satisfied by the exponential itself; requirement 2 is satisfied by dividing by the sum afterwards.

Watch it fix the broken case:

Three genuine probabilities, ordering preserved, no division by zero possible because a sum of positive numbers is always positive.

There is a cost, and it is worth naming. Exponentials grow fast, so softmax exaggerates differences. A logit gap of 1.0 becomes a probability ratio of e^1 = 2.718, so "cat scored one point higher than dog" becomes "cat is 2.7 times more likely than dog". That amplification is deliberate and it is where the "soft max" name comes from: the function is a smoothed version of "pick the largest and give it everything".

Worked example - logits z=[2.0, 1.0, 0.1] for cat/dog/bird: e²=7.389056, e¹=2.718282, e^0.1=1.105171, summing to 11.212509. Dividing each by that sum gives p=(0.659001, 0.242433, 0.098566), which sums to 1.000000, a genuine probability distribution.

Where the three logits come from

This page has been handing you three numbers. Here is a network producing them, so the chain from input to probability is unbroken. Same small rain network as the rest of this module: two inputs, two tanh hidden neurons, now with three output neurons instead of one, because there are three classes.

Read the gradient signs. The two wrong classes get positivegradients, meaning "push these logits down". The correct class gets a negativegradient, meaning "push this logit up". And the sizes are exactly how wrong each one was. dry was already at 2.3%, so its correction is tiny at 0.022533. drizzle at 14.7% gets a firmer shove at 0.146623. Softmax with cross-entropy gives you a correction proportional to the error and nothing else, which is the whole reason this pairing is standard.

One arithmetic check on that gradient: the three components sum to 0.022533 + 0.146623 - 0.169156, which is 0.000000. That is not a coincidence and it happens every time. The probabilities sum to 1, the one-hot label sums to 1, so p - y always sums to 0. Softmax cannot raise one logit without lowering the others; the gradient enforces the same coupling the forward pass has. (The printed six-decimal values happen to cancel perfectly here. They will not always, and if you run this check on other numbers and land on 0.000001 rather than 0.000000 that is the rounding, not a broken gradient. The reason just given is the proof; the addition is only a spot check.)

Shifting every logit changes nothing

Add the same number to all the logits and the probabilities do not move. Not approximately, exactly.

The reason is one line of algebra. Multiplying every exponential by the same constant e^-c multiplies the numerator and the denominator by e^-c, and it cancels. Only the differences between logits ever matter. The absolute level is meaningless, which is another way of saying one of the three output biases is redundant: you could pin it at zero forever and lose nothing at all.

The bars on the stage figure are the probabilities and they always fill the track exactly. That is not a drawing convention, it is the constraint: raising one bar takes width from the others, because there is one track and it is full.

Drag the third slider up on its own and watch the first two shrink without your touching them. Nothing about the first two classes changed. Their scores are identical. Their shares fell because somebody else's share rose.

The top panel is the three raw scores from the break at the top of this page, one of which hangs below the axis. Dividing by the total at that point produces a negative share, and if the three happened to sum to zero it produces nothing at all. The bottom panel is the same three numbers after e^x: every bar is now above the axis, and the ranking strip under each panel is identical, first, third, second, before and after. That is requirement 1 fixed with requirement 3 left untouched, which is exactly what the exponential was chosen for.

This next result is one of the least obvious "nice" identities in this subject. Unlike the sigmoid case on the output-activations page, walking through every step here requires tracking how changing one logit shifts every output probability at once, since softmax ties all the outputs together. We're going to state the result and confirm it numerically, rather than derive it line by line.

Here's a genuine independent check, not a restatement of the formula: nudge just the "cat" logit by 0.001 (z=[2.001, 1.0, 0.1]), recompute softmax and the cross-entropy loss for the true class cat, and measure the change. The measured slope comes out to ≈−0.3409; the predictedgradient, p_cat−1 = 0.659−1 = −0.3410, matches to three decimal places, the same small gap you'd expect from the earlier finite-nudge check, not a coincidence. The general rule, holding for every class at once, is ∂L/∂zᵢ = pᵢ−yᵢ - for this example, the full gradient is [−0.341, 0.242, 0.099].

Why every real implementation subtracts the maximum

Shift invariance is not a curiosity. It is what makes softmax computable.

e^x overflows fast. In standard 32-bit floating point the largest representable number is about 3.4 * 10^38, and e^88 is already 1.65 * 10^38, so e^89 overflows to infinity. Logits in the tens are routine, and it does not take much beyond that to cross the line, so the naive formula produces inf / inf, which a computer reports as NaN(short for "Not a Number", the value it returns when it has been asked for something undefined). NaN then propagates through the rest of the model, poisoning every number it touches, and destroys the run.

Subtracting the maximum makes the largest shifted logit exactly 0, so the largest exponential is exactly e^0 = 1 and nothing can ever overflow. The smallest ones may underflow to 0, which is harmless: they were negligible anyway, and the sum still contains that guaranteed 1, so the denominator can never be zero.

Confirm on the page's own numbers that the trick changes nothing:

This is also the concrete reason the previous page's gotcha exists. When PyTorch's CrossEntropyLoss takes raw logits rather than probabilities, it can do this subtraction and combine the log with the exp internally, cancelling them algebraically instead of computing both. Hand it probabilities and you have already thrown away the numbers it needed.

Softmax over two classes is sigmoid

The last two pages have presented sigmoid and softmax as two different tools. They are the same tool.

Since only logit differences matter, a two-class softmax has one meaningful number in it, the gap between the two logits, and softmax of (z, 0) reduces algebraically to 1 / (1 + e^-z), which is sigmoid. Verify it on any pair with the same gap:

So binary classification with a sigmoid is not a special case that happens to work. It is softmax with the redundant second logit removed, and using one output instead of two saves a parameter that could never have meant anything.

Temperature

One knob, worth knowing because you have already used it. Divide every logit by a number T before applying softmax.

Small T divides the logits by a small number, which spreads them further apart, which the exponential amplifies further still: the top class approaches 100% and the rest approach 0. Large T squeezes the logits together and the distribution flattens toward equal probabilities for everything. In the limits, Tapproaching 0 gives a hard "always pick the winner" and T approaching infinity gives a coin flip among all classes.

The ordering never changes at any temperature, because dividing by a positive number preserves order. Only the confidence changes. This is the same "temperature" setting on every text-generation interface you have used: low temperature makes a model pick its favourite word almost every time, high temperature lets less likely words through.

At inference, you often don't need softmax at all

During training you need the probabilities, because cross-entropy is defined on them. At prediction time, if all you want is "which class", you can skip softmax entirely.

The predicted class is argmax, meaning "the position of the largest number". And since e^x is strictly increasing and dividing by the same positive sum preserves order, the largest logit is always the largest probability. In our example the largest logit is downpour at 1.875071 and the largest probability is downpour at 0.830844. This holds always, not usually.

So a deployed classifier that only reports a label can read the answer straight off the logits. You need softmax when you need to report a confidence, when you need to thresholdon one ("only act if above 90%"), or when you need to compare probabilities across examples. If you only need the label, the extra step buys nothing.