Modern Activation Functions

Every function on this page is trying to fix the same two things about ReLU: its hard corner at zero, and its dead flat negative half.

The corner matters because ReLU's slope jumps from 0 to 1 with nothing in between. Approach x = 0from the left and the slope is 0; approach from the right and it is 1; so there is no single answer to "what is the slope at exactly zero" (the technical phrase is that the function is not differentiable there). In practice frameworks just pick one, usually 0, and nothing catches fire, but a smooth curve avoids the question entirely and gives the training procedure a gentler surface to work on.

The flat half matters for the reason page 4 spelled out: exactly-zero slope means exactly-zero gradient means a parameter that cannot move.

Where these functions differ from Leaky ReLU and PReLU is that they do not fix the flat half with another straight line. They curve. All four of ELU, SELU, GELU and SiLU allow a negative output where ReLU gives a flat zero, but they get there in two genuinely different ways, and the difference is worth having straight before the individual sections.

ELU and SELU slide down to a floor and stay there. As the input goes further and further negative, the output keeps decreasing, more and more slowly, and settles just above a fixed floor: -alpha for ELU, which is -1 at the usual setting, and about -1.7581 for SELU. They only ever go one way, which is what monotonic means. There is no dip and no turning point.

GELU and SiLU do something stranger. Go negative and the output drops to a small negative minimum, then climbs back toward zero as the input gets more negative still. GELU bottoms out at -0.16997 around x = -0.75; SiLU bottoms out at -0.27846 around x = -1.28. The output goes down, then up, so these two are non-monotonic, which is an unusual property for an activation to have. That dip turns out to help, and nobody has a fully satisfying theoretical account of why. This page will say what each one does and what it is measurably good at, and will not pretend the field has more certainty than it does.

ELU

ELU stands for Exponential Linear Unit. For positive inputs it is exactly ReLU: hand the number back unchanged. For negative inputs it follows a curve that starts at 0 and flattens out at a floor of -alpha no matter how negative the input gets. So it never dies (the slope is never zero) and it never runs away (the output cannot go below -alpha), which Leaky ReLU and PReLU both can.

Leaky ReLU's negative branch is still a straight line heading to −∞ for very negative inputs. ELU instead curves and flattens smoothly toward −α as x gets very negative, which keeps a network's average activation closer to zero (like tanh) while its positive side stays identical to ReLU.

SELU

SELU stands for Scaled Exponential Linear Unit, and it is ELU with two very specific numbers glued to it. The idea is unusual: instead of choosing an activation and then adding a separate normalization layer to keep the numbers in each layer well behaved, pick the activation's constants so that the normalization happens automatically. "Well behaved" here means each layer's outputs have mean 0 (they average out to zero rather than drifting up or down) and variance 1 (they are spread out by a consistent amount, neither collapsing to nothing nor exploding). Those two constants are not tuned or approximate. They are the exact solution of the equations that force that property.

SELU uses α≈1.6733 and λ≈1.0507, chosen so that if a layer's inputs already have mean 0 and variance 1, its outputs land back at mean 0 and variance 1 too, layer after layer. The activation function does its own normalizing, without needing a separate technique like Batch Normalization.

GELU

GELU stands for Gaussian Error Linear Unit, and it needs one piece of vocabulary first.

Phi(x)is a gate that behaves a lot like sigmoid: it is always between 0 and 1, and it climbs smoothly from "mostly off" to "mostly on" as x increases, so it looks a great deal like sigmoid without being sigmoid. That is the whole working picture, and it is enough to use GELU correctly.

For the curious, Phi(x) has a formal name too: the standard normal cumulative distribution function, which is a long name for a simple question about the "bell curve" shape that shows up whenever a lot of independent random effects average out (heights in a population, measurement error, and so on): if you draw one random number from that standard bell curve, what is the chance it comes out below x? The bell curve is centred at 0, so Phi(0) = 0.5 exactly, half the curve is below zero. Phi(1.4) = 0.919243, meaning 91.9% of the bell curve sits below 1.4. Phi(-0.7) = 0.241964, meaning only 24.2% sits below -0.7. None of this changes how GELU behaves - it is background for readers who want to know where the curve comes from, not a requirement for using it.

GELU is then x multiplied by that number: GELU(x) = x * Phi(x). Where ReLU makes a hard yes-or-no decision about whether to keep a value, GELU keeps a fraction of it, and the fraction is how confidently positive the input is. At x = 1.4 you keep 91.9% of it. At x = -0.7you keep 24.2% of it, and since the value itself is negative, you get a small negative output instead of ReLU's hard zero. At x = 0 you keep exactly half of zero, which is zero, so the two functions still agree at the origin.

GELU is the default activation in most modern transformer architectures (BERT, GPT-style models), though those are beyond this module's scope. Notice it is non-monotonic: it dips slightly negative for small negative x, unlike ReLU's hard 0, before rejoining x for large positive x.

Swish / SiLU

SiLU stands for Sigmoid Linear Unit, and Swish is the same function with an extra tunable knob. It is the same idea as GELU with a cheaper gate: instead of Phi(x), multiply x by sigmoid(x). Since sigmoid also runs from 0 to 1 and also climbs smoothly, it does almost the same job for a fraction of the compute, which is why the two functions look nearly identical when plotted. Swish(x) = x * sigmoid(beta * x) adds a beta that controls how sharply the gate opens; setting beta = 1 gives SiLU, and that is what nearly everyone uses.

Like GELU, it dips slightly below zero for small negative inputs before flattening out, which turns out to help gradient flow in deep networks.

Softplus

Softplus is the answer to "what if ReLU's corner were rounded off". It rises smoothly from just above 0, bends through ln(2) = 0.693147 at the origin, and for large positive inputs becomes indistinguishable from x itself. It never returns exactly zero for any input, so no neuron can ever die. The catch is in that same sentence: it never returns exactly zero, so a neuron that ought to be silent still leaks a little signal, and you lose the genuine sparsity (large blocks of exact zeros) that makes ReLU networks cheap.

Maxout

Maxout is the odd one out on this page, and the odd one out in the whole module. Everything else here is a fixed curve applied to a number that a neuron already computed. Maxout replaces the curve entirely: run k separate weighted sums on the same inputs and keep whichever came out largest. There is no curve to apply, because the shape of the resulting function is whatever the learned weights make it - at the cost of k times the parameters for that layer.

The exact shape depends entirely on the learned weights, which is the whole point.

One maxout unit, built from the rain network

Here is Maxout built out of the small rain network's own two hidden neurons, which makes the parameter cost concrete. Take h1 and h2 exactly as they are, and instead of activating each one separately and passing both forward, keep only the larger.

Six parameters in, one number out. The same six parameters used as two ordinary neurons give you two numbers out. That is the "k times the parameters" cost, counted: a Maxout layer of width m with k = 2 needs as many weights as an ordinary layer of width 2m.

What you buy is that the unit's shape is not fixed in advance. On one side of the line 2.50 * x1 + 1.50 * x2 = 0.80 it behaves like one linear function, on the other side like a different one, and where that boundary sits is learned. ReLU is the special case where piece B is pinned to the constant 0 and never learns.

The bold envelope is what the neuron outputs, and the kink in it is where the two pieces swap over. That kink's position is learned, which is the whole of Maxout.

The modern family, one network

Same comparison as the previous page, same network, same input, so the two tables can be read side by side. Inputs x1 = 0.80, x2 = 0.60; hidden pre-activations z1 = 1.40 and z2 = -0.70; output weights v1 = +1.20 blue, v2 = -0.80 orange, bo = +0.30.

The arithmetic behind two of those rows, written out, since the rest follow the same pattern:

Three things fall out of the table.

GELU lands almost exactly where ReLU lands. 0.8787 against 0.8787, agreeing to four decimals by coincidence of this particular input, but the near-agreement is not coincidence: GELU is a smoothed ReLU and on a positive-leaning input the two barely differ. This is the honest picture of the modern family. They are refinements, not revolutions.

SELU's output is the outlier, and that is expected. It multiplies everything by lambda = 1.0507 and pushes the negative branch down as far as -0.8851, so its final probability is 0.9412, the highest on the page. That is what the scaling is for. It also means SELU is not a drop-in swap; changing to SELU changes the scale of everything downstream.

Softplus is the only row with a positive a2. 0.4032, not negative. Softplus never outputs a negative number, so like sigmoid it is not zero-centred, and the zero-centering complaint from the previous page applies to it too.

Everything separating these six is on the left of the vertical axis. Probe the right-hand side and ELU, SELU minus its scaling, GELU and SiLU are all converging on the same straight line that ReLU already was. So: probe left, and read four different behaviours.

ELU heads for -1.0000 and stops. At -3 it is at -0.950213, at -4 it is at -0.981684, which is 98.2% of the way to its floor. It never crosses it, and it never turns round.

SELU is the same shape stretched. Its floor is -1.758099, and at -4 it reads -1.725899, which is 98.2% of the way to its floor, the identical fraction, because SELU is ELU scaled by two constants and scaling does not change where a curve sits along its own journey.

GELU dips and returns. The lowest it ever gets is -0.16997, at about -0.752, and by -3 it is back to -0.004050, which is visually on the axis.

SiLU dips deeper and returns more slowly. Its minimum is -0.27846 at about -1.278, and at -3 it is still at -0.142278, a good deal further from zero than GELU is at the same point.

Softplus never goes below zero at all. At -0.70 it reads 0.403186 and at -4 it reads about 0.018. It approaches the axis and never touches it, which is the property that makes it immortal and non-sparse in the same breath.

Now the bottom panel, and one entry in it is worth stopping on before rung 4 gets to it: probe 1.40 and read GELU's slope of 1.129. No curve on the previous page ever exceeded 1.

A common fast approximation to GELU, used in real code (BERT, GPT-2):

This is a fast approximation to the true erf-based formula above, not an alternate definition - some frameworks default to it, some to the exact version.

Two entries in that table are worth stopping on.

GELU's slope at 1.40 is 1.1289, greater than 1. No function on the previous page ever exceeded a slope of 1. GELU's does, over a band of positive inputs, because it is catching up to x from below as Phi climbs toward 1. A slope above 1 slightly amplifies the training signal passing through rather than shrinking it, which is the opposite of the vanishing-gradient problem.

Softplus's slope is exactly sigmoid. 0.8022 at z = 1.40 and 0.3318 at z = -0.70 are precisely sigmoid(1.40) and sigmoid(-0.70), the same two values in the sigmoid row of the previous page's activation table. That is not a numerical accident: the derivative of ln(1 + e^x) is e^x / (1 + e^x), which is exactly sigmoid(x). It is the same curve you met on the previous page.

Verifying SELU's central claim

The page claims SELU's two constants are chosen so a layer's outputs come back to mean 0 and variance 1. That is checkable, so here it is checked.

The condition the constants have to satisfy is: if the numbers going into SELU are spread out like a standard bell curve (mean 0, variance 1), then the numbers coming out must have mean 0 and variance 1 as well. Computing those two quantities for alpha = 1.6732632 and lambda = 1.0507010 gives:

Averaging to 0, with an average squared size of 1, is exactly what "mean 0 and variance 1" means. Both land on the target to six decimal places, which is what "solved the fixed-point equations" means in practice. Those two conditions are two equations in two unknowns, and alpha and lambda are their solution. Nothing was tuned.

SELU is a package deal, not an activation you can drop into an existing network. It delivers the behaviour above only if the rest of the network is set up its way, and that means two specific things: a particular way of choosing the starting weights, and a particular variant of one common training trick. Use the ordinary version of either and the guarantee is gone, leaving you with an oddly-scaled ELU and none of the promised self-normalization. For completeness, the two conditions are initialization from a normal distribution with variance 1/n, where n is the number of inputs to the layer, a scheme usually called LeCun normal, and alpha dropout rather than ordinary dropout. Both are covered properly on the initialization and training-time-regularization pages. You do not need either term yet. You only need to know that SELU comes with strings attached.

The SELU bars hold near the L0 height layer after layer; drag the init-scale slider off 1.0 and watch that stop being true, which is this rung's point made visible.

Which of these should you use

The honest version, which is shorter than the length of this page implies.

GELU if you are building or fine-tuning a transformer, because that is what the architecture was tuned around and matching it removes a variable.

SiLU / Swish as a general-purpose smooth alternative to ReLU. It is cheaper than exact GELU and behaves almost identically.

ELU if you specifically want the bounded negative branch, most often in smaller networks without batch normalization.

SELU only if you commit to the whole package: LeCun normal initialization, alpha dropout, and a plain fully-connected stack. Outside those conditions it is just an oddly-scaled ELU.

Softplus rarely as a hidden activation. Its main modern use is on an output where the answer must be positive, such as predicting a variance or a count, which the output-activations page picks up.

Maxoutrarely, because doubling a layer's parameters is a steep price and batch normalization solved most of what it was proposed for.

And the thing worth saying plainly: on most ordinary tasks the gap between ReLU, GELU and SiLU is small, often smaller than the run-to-run variation from a different random seed. The table above moved the final probability from 0.8622 to 0.9412 across the whole family, which looks large, but that is one hand-set network on one input with no training to compensate. A trained network adjusts its weights around whichever activation it was given. Activation choice is worth getting sensible and rarely worth agonising over.