Layers and the Forward Pass

A single neuron can only draw one straight dividing line through its inputs - it can decide "more of clue A, less of clue B" but it can't represent an "either/or" kind of pattern (bring an umbrella if it's cold orif it's windy, but not if it's both, say). This particular kind of pattern, true when exactly one of two things holds, false otherwise, has a name: XOR("exclusive or"), and it's the textbook example of something one neuron provably cannot learn, no matter how its weights are set. Real problems are rarely that simple. The fix is the same one your brain uses: don't rely on one neuron, chain many of them together.

A layer is a row of neurons that all read the same set of numbers and all produce their own answer. That is the whole definition. The neurons in a layer do not talk to each other; they work in parallel on identical input and disagree, which is the point.

Three pieces of vocabulary that get used constantly from here on:

Width is how many neurons a layer has. Depthis how many layers the network has. A network described as "2 inputs, 2 hidden, 1 output" is two wide and one hidden layer deep.

Fully connected (also called dense) means every neuron in a layer reads every number from the previous layer. It is the default arrangement, and the only one this module uses. Convolutional layers, later in the curriculum, deliberately break it.

Counting parameters is worth doing once by hand, because it is how you size a model. Every connection is one weight, and every neuron has one bias. For a layer with n inputs and m neurons, that is n * m weights plus m biases. For the small network used throughout this page: the hidden layer has 2 inputs and 2 neurons, so 4 weights and 2 biases; the output layer has 2 inputs and 1 neuron, so 2 weights and 1 bias. Nine parameters total. Every one of those nine is a number that training is allowed to change, and nothing else about the network changes at all.

That claim ("provably cannot", not just "hasn't managed to") deserves the proof, because it is four lines of arithmetic and it is the reason deep learning exists.

Take one neuron with two inputs, and say it answers yes when z > 0. XOR demands these four answers:

The last two lines contradict each other, so no values of w1, w2 and bcan satisfy all four demands at once. Not "no values anyone has found". No values exist. This is the geometry from the previous page cashed out: one neuron draws one straight line, and no straight line puts (0,1) and (1,0) on one side while putting (0,0) and (1,1) on the other. Draw the four points on paper and try it.

And here's the two-layer network that does solve it

The fix takes exactly one hidden layer of two neurons, with ReLU as the activation. ReLU is max(0, z), meaning "keep the number if it is positive, otherwise give zero". The next page covers it properly; for now that one line is all you need.

Four for four. Now look at what the two hidden neurons actually do, reading their values straight out of the block above rather than guessing.

h1 counts how many inputs are on. It outputs 0, then 1, then 1, then 2. It is not a yes/no answer at all; it is a tally.

h2 only wakes up when both inputs are on. It outputs 0, 0, 0, then 1. Its bias of -1 is what holds it shut until the tally reaches 2.

The output neuron then computes h1 - 2 * h2: take the tally, and if h2 reports that both are on, subtract twice as much as h1 added. Follow the last row: h1 contributes 2, h2triggers a subtraction of 2, and the total is dragged back to 0. So the output reads as "how many inputs are on, minus a correction that switches on precisely when there are too many". That is "one but not both", which is XOR.

It is tempting to call h1 an OR gate and h2 an AND gate, and that is nearly right but not right: a real OR gate would output 1 at input (1,1), and h1 outputs 2. The correction term is doing the work that the extra 1 creates. Getting this exactly right matters, because the whole point of the block above is that you can check it.

Neither hidden neuron is doing anything a single neuron couldn't do. A tally and a both-are-on detector are each a single straight line and always were. What the second layer buys is the ability to combine two straight-line answers into something that is not a straight line, and that combining is the entire reason depth works.

These particular nine numbers were chosen by hand so the arithmetic comes out in whole numbers. A trained network would find different ones, almost certainly uglier ones, that also work. This exact solution is the one in Goodfellow, Bengio and Courville's Deep Learning, section 6.1.

The previous page said the activation function is there to bend the neuron's response. Here is that claim as arithmetic, using the small rain network that the rest of this module reuses.

Now run it with no activation at all, meaning every neuron just passes its pre-activation straight through.

Two layers, three neurons' worth of arithmetic, nine parameters, answer 2.540000. (Three, not five: the two inputs are just numbers sitting there, and an input layer computes nothing.) Now here is a single neuron that gives the identical answer:

Identical, to the last digit, and not by coincidence for this input. That single neuron produces the same answer as the whole two-layer network for every possible input, because a weighted sum of weighted sums is just another weighted sum. Stack a thousand layers this way and the same folding collapses all of them into one neuron with one straight boundary. You would have a thousand times the parameters and exactly none of the extra power.

The activation function is the one thing that stops the folding. Take tanh, which is the curve this module uses in its hidden layer from here on: it is an S-shaped squashing curve like the sigmoid from the previous page, but it runs from -1 to 1 instead of 0 to 1, so a neuron that concludes "no" is allowed to say so with a negative number. Once a1 = tanh(z1) sits between the two layers, there is no set of weights W1, W2, B that reproduces the network, because tanh is not a weighted sum of anything. That, and not tidiness of range, is why step 3 exists.

The instrument draws this rung for you. Set its hidden activation to Linear and a panel appears under the readout printing W1, W2 and B live from whatever the nine sliders currently hold, plus the check line. Switch back to Tanh and the panel is replaced by one sentence saying no such neuron exists. The fold appearing and disappearing as you flip one dropdown is the picture.

Now the real thing, end to end, with tanh in the hidden layer and a sigmoid on the output because the question is yes-or-no. Recall from a moment ago what tanh is: the same S-shape as sigmoid, running from -1 to 1 rather than 0 to 1, so it can return a negative number. That is the only thing you need to know to follow the block; page 4 covers why it is the better choice inside a hidden layer. Watch a2 come out negative below and you will see the range doing its job.

The answer is not the 80.2% from the previous page, and it should not be. Same weather, same two clues, same first neuron, different number. Page 2 was one neuron deciding on its own. This is two neurons disagreeing and a third weighing them up, through a different curve, which is a different and better machine. A bigger network giving a different answer from a smaller one is the normal case, not a mistake, and it is the whole reason to build the bigger one. Watch for this again on the next two pages, where changing only the hidden activation moves the same answer between 73.1% and 94.1%.

Three more things worth noticing, because they are the things that generalise.

The two hidden neurons disagree, and that is the point. h1 came out strongly positive at +1.40, h2 strongly negative at -0.70. They read the identical two inputs and reached opposite conclusions, because they hold different weights. A layer is useful precisely to the extent its neurons disagree; if they all held the same weights they would all produce the same number and the layer would be one neuron wearing a costume. That failure has a name, symmetry, and the weight-initialization page is about avoiding it.

The negative weight does real work. h2 ends up at -0.604368, and the output neuron reads it through v2 = -0.80, drawn orange. Negative times negative gives +0.483494, a positive contribution. h2is "evidence against rain" being read by a connection that treats evidence-against as evidence-for. Follow the two signs and it makes sense; guess at it and it won't.

The activation is applied one number at a time. tanh did not look at z1 and z2 together. It ran on each independently. Every activation in this module works that way, with exactly one exception: softmax, on the last page, which deliberately couples all the outputs together. Everything else is element by element.

Writing out every multiplication is how you learn it and not how anyone codes it. Here is the identical calculation in the form you will actually meet.

Stack the hidden layer's weights into a grid with one row per neuron:

One trap, worth catching now rather than in someone else's code. The entry in row 1, column 2 of that grid is +2.00, which is the weight this module has been calling w21. So the grid puts the neuron first and the input second, which is the exact reverse of the w<input><hidden>subscript order you have been reading all page. Nothing is wrong with either; both orders are standard, and which one a piece of code uses is a convention its author picked. Read the grid as "one row per neuron" and the subscripts as "input, then neuron", and check which one you are looking at before trusting an index.

Then the entire hidden layer is z = W x + b, where multiplying a grid by a column of inputs means "take each row, dot it with the input, and write the result on its own row":

Same two numbers as above, same arithmetic, one symbol. Then a = tanh(z) applied to every entry, and the output layer is zo = V a + bo with V = [ +1.20 -0.80 ].

Two full layers of a neural network:

That is the same arithmetic you just did by hand, with nothing hidden and nothing added. Matrix notation is a shorter way to write many multiplications, not a different algorithm. The backpropagation page makes the same point about the backward pass.

How far can this go?

There's a real theorem here, not just an intuition: a network with even a single hidden layer, given enough neurons in it, can approximate any continuous function on a bounded input range to any accuracy you like - this is called the universal approximation theorem. It's worth reading carefully, though, because it's easy to overclaim: it says such a network exists, not that gradient descent will actually find it, and not that "enough neurons" is a small or practical number. In practice, going deeper (more layers) tends to represent complex functions far more efficiently than going wider with one giant layer, which is the whole reason the rest of this module, and deep learning generally, is about depth rather than a single very wide layer.

One practical footnote to that theorem. It says a wide-enough single hidden layer suffices, and it is silent about howwide. For many functions the required width grows exponentially with the input dimension, while a deeper network can express the same function with a number of neurons that grows far more gently. That gap is why "just make one layer bigger" is not the answer in practice, and why the field is called deep learning rather than wide learning.