Output Activations

A hidden neuron's output only ever feeds more of your own network, so its range can be whatever's computationally convenient. An output neuron's number has to mean something to whoever reads it, "70% chance of rain," not an arbitrary number, and that meaning should decide its activation. The raw, pre-activation number a neuron computes is usually called a logitonce it's about to be fed into a probability-producing function like sigmoid or softmax. "Logit" just means the number before the squashing.

Two rules underneath everything on this page.

The activation must be able to produce every answer the task allows. If rainfall can be 40 mm, an output squashed into 0-to-1 can never say so, no matter how it is trained. If a probability can never be negative, an output that ranges over all real numbers is inviting nonsense.

The activation and the loss are chosen together, not separately. Sigmoid with binary cross-entropy behaves well for a reason the derivation in rung 4 makes exact. Sigmoid with squared error behaves badly for the same reason when the target is a hard 0-or-1 class label, because that is the case where the network can end up confidently wrong, pinned deep in sigmoid's flat region where the slope is nearly zero, and squared error has no way to shout at it. When the target is a genuine fraction sitting somewhere in the middle, like 0.35 of a tank, the network is never pushed into the flat tails and the trap does not spring. Hard class labels, use cross-entropy. Interior fractions, squared error is fine.

Everything below runs on the same network the rest of this module uses, so only the last layer changes. Inputs x1 = 0.80 and x2 = 0.60, one hidden layer of two tanh neurons, giving:

Those two numbers, a1 = 0.885352 and a2 = -0.604368, are the hidden layer's summary of the weather. Everything from here is three different ways of turning them into an answer.

Head 1 - regression

The question: how many millimetres of rain will fall.

The answer can be 0.4 mm or 40 mm. It has no ceiling, so it must not be squashed. The output neuron computes its weighted sum and that number isthe prediction. This is usually called a linear output, or "no activation", and both mean the same thing: the identity function, f(z) = z.

Two symbols in that last block are new, and both are read out loud more easily than they look.

dL/dzo is said "d L by d z-o", and it means "how much does the loss L change when zo changes". The L on top is the thing being affected, the zounderneath is the thing being wiggled. It is the slope idea from the activation pages pointed at a different pair of quantities: instead of "how much does this curve's output move when its input moves", it is "how much does our total wrongness move when this one number moves". A big dL/dzo means zo has a lot of influence over the loss. A dL/dzo of zero means moving zo would not change the loss at all. That quantity, for a parameter rather than for zo, is exactly the gradient page 1 described. It is also the quantity the backpropagation page calls δ; the two names are for one thing, and this page happens to be looking at it at the very end of the chain rather than in the middle.

f'(zo) is said "f prime of z-o", and it is the same thing the activation pages measured: the slope of the output activation f, at the particular point zo. Here the output activation is the identity, whose slope is 1 everywhere, which is why the line reads * 1.

Put together, the line says: how wrong you were, scaled by how much the final curve was willing to pass along. Both symbols recur on this page and the next, and they never mean anything else.

The slope of a linear activation is exactly 1 everywhere, so it never shrinks the training signal and never blocks it. The gradient arriving at the output layer is just "how far off you were, and in which direction". Negative means the prediction was too low.

Do not use cross-entropy here. Cross-entropy takes the logarithm of the prediction, and this prediction is not a probability; it can be 40, and it can be negative, and log of a negative number does not exist. Squared error is the right pairing for an unbounded output.

Two variants, when the answer is constrained:

Softplus is generally the better choice than ReLU on an output that must be positive, for the reason the activation-functions page established: ReLU's exactly-zero region gives an exactly-zero gradient, and an output neuron stuck there stops learning entirely.

Head 2 - binary

The question: will it rain, yes or no.

The answer must be a probability, so it has to sit strictly between 0 and 1. Sigmoid is the standard choice, and the same weighted sum from above is what gets squashed. That raw sum, 1.845916, is the logit.

Notice that the two rows share a weighted sum and differ only in what is done to it. The network's internals did not change at all between the regression head and this one; only the final step and the loss did.

Notice also the sign of the gradient in each case. When it rained and the network said 86%, the gradient is -0.136353: push the logit up, be more confident next time. When it did not rain, the gradient is +0.863647: push the logit down, hard. The size of the gradient is exactly how wrong the probability was, which is the cancellation rung 4 explains.

Head 3 - multi-class

The question: dry, drizzle, or downpour. Exactly one is true.

Now the output layer needs three neurons, one per class, each with its own weights into the same two hidden activations. Three logits come out, and softmax turns them into three probabilities that add to 1.

The block below calls these output weights u. That is only a label: w has been the input-to-hidden weights all module, vwas the single output neuron's weights a moment ago, and ukeeps this head's three sets apart from both. They are ordinary weights and there are six of them plus three biases.

Three separate weighted sums, three separate sets of weights, one shared hidden layer. That is what "one output unit per class" means in practice.

Softmax is the one activation in this entire module that is not applied one number at a time. p_dry depends on z_downpour, because z_downpour is in the denominator. Push one logit up and every other probability goes down, automatically, because they are forced to sum to 1. That coupling is the point, and it is what the next page is about.

The same three logits, run both ways

The three numbers below are not from the rain network, because the rain network does not tag photographs. Take them as any three logits a photo-tagging network might produce; where they came from does not matter here, because the point is entirely about what the two treatments do to the same three numbers.

Look at what softmax did to the first label. Read independently, the network is 76.9% sure the photo is outdoor. Run through softmax, that becomes 29.3%, not because the evidence changed but because it had to give room to "people". Softmax made the labels compete when they were never in competition.

The loss follows the same split. For multi-label, run binary cross-entropy on each label separately and add up the results. With true labels y = (1, 0, 1), meaning outdoor and people are correct and night-time is not:

Three independent yes/no problems that happen to share a network, which is exactly what multi-label is.

On the stage figure above: the two circles on the far left are a1 = 0.885352 and a2 = -0.604368, the hidden layer, frozen. The dashed circle in the middle is that whole layer treated as one object, because from here on nothing about it matters except the two numbers it produced. The circle on the right is the logit, and it is the only thing you control. The activation is not drawn as a circle at all; it is the arrow leaving that circle, which is the right way to picture it, because the activation is a step and not a place.

The top two branches carry the identical logit, 1.845916, drawn identically because it is identical: the regression head and the binary head compute the same weighted sum and differ only in what they do to it afterwards. The bottom branch is the one that needed a different output layer, because a question with three answers needs three logits, and those three are the only numbers on the figure that are forced to add up to anything.

Both symbols in the derivation below were read out loud in rung 2 and neither means anything else anywhere on this site: dL/dz is how much the loss moves when z moves, and s(1-s)is sigmoid's own slope at the point it is sitting at.

The derivation leans on one calculus rule, d/ds[log(s)] = 1/s, so let's verify it first with the same nudge-and-measure method used on the earlier pages: at s=0.5, log(0.5)=−0.6931; at s=0.501, log(0.501)≈−0.6911; the change is 0.0020 over a nudge of 0.001, so slope≈2.0, matching 1/0.5=2.0. ✓

Now the full derivation, in three explicit steps, for a sigmoid output s=σ(z) with binary cross-entropy L=−[y·log(s)+(1−y)·log(1−s)]:

That cancellation is the whole point. Compare a confidently wrong prediction (y=1, s=0.02) under each loss: cross-entropy gives ∂L/∂z = 0.02−1 = −0.98, a strong signal to correct course. Squared error, with no cancellation to save it, gives ∂L/∂z = (0.02−1) * 0.02 * 0.98 = −0.0192, nearly 50 times smaller, exactly when the network is most wrong and most needs a push.