Activation Functions

An activation function is a plain function: one number in, one number out. It has no weights, learns nothing, and knows nothing about the rest of the network. When a layer of 500 neurons applies ReLU, that is 500 completely independent applications of max(0, z), one per neuron, with no communication between them. The single exception in this whole module is softmax on the last page, which deliberately looks at all the outputs at once.

There is also exactly one activation here with something to learn, PReLU, which owns a single trainable slope. Every other function on this page and the next is fixed for the life of the network.

Which one you pick changes three things: the range the neuron's output can take, how steep the function's slope is, and whether that slope can ever hit exactly zero and strand the neuron. Two of those three are about slope, so slope needs pinning down before the table arrives.

You already know what the slope of a straight line is: go one step right, and slope is how far you went up. Every curve on this page bends, so that single number is not available. A curve has a different slope at every point on it. Measuring it is something you can do by hand, and the method is the definition. Pick a point. Nudge the input by a hair. See how much the output moved. Divide. Rung 2 does exactly that, with numbers.

Sigmoid

The original. It takes any number, however large, and squeezes it into the range 0 to 1, gently, with no hard edges. A very negative input comes out near 0, a very positive one near 1, and 0 comes out at exactly 0.5. That makes its output directly readable as a probability, which is why it survives to this day on the output layer of every binary classifier, and it is the same function that turns a neuron into logistic regression.

Its problem is what happens away from the middle. At z = 4, sigmoid gives 0.982014; at z = 5, 0.993307. A whole unit of input bought about one percent of output. The curve has gone flat, which is called saturation, and a flat curve has a near-zero slope. Sigmoid's slope maxes out at 0.25, at the single best point on the entire curve, and falls away in both directions. Slopes multiply along the chain, so five sigmoid layers can shrink a training signal by 0.25^5 = 0.000977 at absolute best, and much worse in practice, because 0.25 is the best case and most neurons are nowhere near it. That is the vanishing gradient problem, and it is why sigmoid was displaced from hidden layers.

Tanh

The same S-shape as sigmoid, but stretched to run from -1 to 1 and centred so that tanh(0) = 0 exactly. That centring is the entire reason to prefer it in a hidden layer, for the reason spelled out in the zero-centering note below.

It is also steeper where it counts. At its best point its slope is 1.0 against sigmoid's 0.25, exactly four times as steep, and it stays ahead of sigmoid for any zbetween about -1.66 and +1.66, which is where a well-behaved network's pre-activations mostly sit. Push further out than that and the advantage reverses: tanh saturates faster than sigmoid, and by z = 5sigmoid's slope is around 36 times tanh's. So tanh is not uniformly better, and it is worth being exact about that, because it explains the next sentence instead of sitting in tension with it. Tanh saturates too. It just starts from a much better height, and it flattens on both sides rather than only pinning outputs to one end. Tanh remains a reasonable default for small networks and is what this site's playground uses.

ReLU

max(0, z). If the number is positive, hand it back unchanged. If it is negative, hand back zero. That is the whole function, and it is a strange thing to be the most important activation in deep learning, because it is barely a function at all.

It wins on the derivative column. For any positive input the slope is exactly 1, not 0.25, not 0.9, exactly 1, forever, no matter how large the input gets. A training signal passing back through a positive ReLU is multiplied by 1 and comes out unchanged. Chain fifty of them and the signal is still intact. That single property is what made networks deeper than a handful of layers trainable at all, and it is why ReLU is still the default hidden activation you should reach for first. It is also almost free to compute: one comparison, no exponentials.

Its failure mode is the flat half. For any negative input the slope is exactly 0, and zero is a hard number. A neuron whose pre-activation has drifted negative for every example in your data receives a gradient of exactly zero on every one of them, so its weights never change, so its pre-activation stays negative. It is not learning slowly, it has stopped. The name for this is a dead neuron, and the next two rows of the table exist entirely to prevent it.

Leaky ReLU

ReLU's flat zero region for negative inputs means a neuron that ever lands entirely in negative territory gets gradient exactly 0, forever - it "dies" and stops learning. Leaky ReLU tilts that flat floor into a very shallow ramp instead, so there's always some gradient available to recover. Worked: at x=−3, f(x) = 0.01 * (−3) = −0.03 (compare ReLU's hard 0 here). Nudge check confirms the slope in the negative region is exactly α=0.01.

PReLU

Same shape as Leaky ReLU, but instead of hand-picking α, the slope ais a parameter the network learns via gradient descent, just like every weight and bias, typically initialized at a=0.25 (the paper's chosen starting value, not a mathematical requirement). Worked: at x=−2, a=0.25 → f(x)=−0.5. During training, a gets nudged up or down by the same process that updates every other parameter: the same nudge-and-measure test used above for z, applied to a instead - for x≤0, nudging a by a hair moves f(x) by that same hair times x, and for x>0 it moves it not at all. Exactly how that nudge gets computed and applied is covered in full on the backpropagation page; for now, the point is just that PReLU's slope is learned, not hand-picked.

Optional aside - zero-centering

Sigmoid's output is always positive, somewhere between 0 and 1, never negative. That means the gradient signal flowing backward through a sigmoid neuron always carries the same sign as that neuron's own upstream error, which tends to push all of that neuron's incoming weights in the same direction on any given update, even when some of them really ought to move in opposite directions, a slow, zig-zaggy way to train. Tanh, being centered on zero, doesn't have this problem, which is one reason it's often preferred over sigmoid for hidden layers in practice.

That 0.159 is the slope of sigmoid at the single point z = 1.40. Its formal name is the derivative, and it is written f', said "f prime": f'(1.40) = 0.159. Nudge at a different point and you get a different number, which is the entire premise of this page. A steep spot means a small change going in produces a big change coming out. A flat spot means almost nothing happens no matter what you do.

Five activations, one network

Now the comparison that matters. Same network, same input, same weights, five different hidden activations, traced all the way to the output. This is the small rain network from the previous page: inputs x1 = 0.80, x2 = 0.60, hidden weights (+1.50, +2.00, b = -1.00) and (-1.00, +0.50, b = -0.20), output weights v1 = +1.20 blue, v2 = -0.80 orange, bo = +0.30.

The hidden layer's two pre-activations do not depend on which activation you chose, so they are the same in every row:

Everything after that point differs.

Read it column by column and the table teaches the whole page.

Column a1, the positive input. ReLU, Leaky ReLU and PReLU all hand back 1.4000 untouched, because for positive inputs all three are literally the identity function. Sigmoid crushes 1.40 down to 0.8022 and tanh to 0.8854. The squashing functions are throwing away magnitude information that the ReLU family keeps.

Column a2, the negative input. This is where all five rows separate, and the numbers are worth reading carefully. ReLU gives exactly 0.0000: h2's contribution is deleted. Leaky ReLU gives -0.0070, which is 0.01 * -0.70, technically alive but contributing almost nothing. PReLU with a = 0.25 gives -0.1750, a real signal. Tanh gives -0.6044, the largest number in the column by a distance, because tanh is the only function here that maps a moderately negative input to a strongly negative output rather than shading it toward zero. Sigmoid gives +0.3318, which is positive, because sigmoid's output is positive for every input including very negative ones. A neuron that concluded "no" still emits a positive number, which is precisely the zero-centering complaint in rung 1.

Column p. From 73.1% to 89.3% on identical evidence. Changing nothing but the hidden activation moved the network's final probability by 16 percentage points. This is not a small architectural detail.

Look at the bottom panel first, and ignore the top one for a moment.

Sigmoid's slope curve is a low hill. It peaks at 0.25 directly above x = 0 and falls away symmetrically. Everything to the left of -2 and to the right of +2 is nearly touching the axis. That whole flat region is where a saturated neuron lives.

Tanh's slope curve is the same shape, four times taller. It peaks at 1.00, also above x = 0. And it is narrower, which is the thing the picture shows and prose gets wrong: past about x = 1.66 the two curves cross and sigmoid's is the higher one from there out. Drag the probe to 3.00 and read it: sigmoid 0.045, tanh 0.010.

ReLU's slope curve is not a curve. It is a floor at 0.000 for every negative x and a ceiling at 1.000 for every positive one, with a vertical jump between them. There is no hill and no falling away. That flat 1.000 stretching off to the right, never sagging however far out you probe, is the entire reason ReLU replaced the S-curves.

Leaky ReLU and PReLU are ReLU with the floor lifted off the axis. At the default a = 0.25, PReLU's negative floor sits at 0.250, which is exactly sigmoid's best-case slope, available on the whole negative half rather than at one point. Leaky ReLU's sits at 0.010 and you have to look closely to see it is not zero. It is not zero, and that is the whole of the difference between a neuron that can recover and one that cannot.

Now the top panel, which is the one people look at first and which matters less. Its only job is to tell you the range: sigmoid between 0 and 1, tanh between -1 and 1, the ReLU family unbounded above. Note that three of the five curves in the top panel are literally the same straight line for every positive x. Nothing distinguishes ReLU, Leaky ReLU and PReLU on the right-hand side at all; everything that separates them is on the left, and most of it is in the bottom panel.

Now the fact that makes slope the most important column in the table. It is borrowed from the backpropagation page, which derives it properly, and it is stated here rather than proved because everything on this page and the next depends on it:

To work out how a weight in an early layer affects the final loss, the network multiplies together the slope of every step in between. Not adds. Multiplies. A weight five layers from the output has its influence routed through five slopes, and the gradient that arrives back at it is the product of all five.

That single sentence is why a slope is not a piece of trivia about a curve's shape. Multiply five numbers that are each at most 0.25 and you get at most 0.000977, so the signal reaching that early weight is a thousandth of what left the output, and the weight barely moves. Multiply five numbers that are each exactly 1 and the signal arrives intact. Multiply anything at all by a single 0 and it is gone permanently. Those three sentences are the whole argument of the next four pages, and every one of them is a consequence of slopes multiplying.

Which one should you actually use

Short version, and it really is this short for hidden layers.

Start with ReLU. It is the default for a reason: fastest to compute, no vanishing-gradient problem on the positive side, and it works. Most networks never need anything else.

Move to Leaky ReLU or PReLU if neurons are dying. The symptom is a chunk of your hidden layer outputting exactly 0 for every input in your data. Leaky ReLU costs nothing and removes the failure mode. PReLU adds one learnable number per neuron and occasionally helps more, at the cost of one more thing that can overfit.

Use tanh for small networks, or where you need outputs centred on zero. It is what this site's playground uses, because for a two- or three-layer network the vanishing problem never gets a chance to bite and the zero-centred output makes the visualisations honest.

Do not use sigmoid in a hidden layer.Its 0.25 maximum slope and its always-positive output are both liabilities, and tanh is the better of the two for that job over the range a healthy network actually operates in. (Not over every possible input, as the tanh paragraph above is careful to say, but the tails where sigmoid wins are tails you do not want your neurons sitting in anyway.) Sigmoid's place is the output layer of a binary classifier, which is a different decision entirely and gets its own page.

That last distinction is important enough to state plainly: the hidden activation and the output activation are two separate choices. Nothing forces them to match. A network with ReLU throughout its hidden layers and a sigmoid on the output is completely normal, and is in fact the most common binary classifier there is.