The Single Neuron

Let's build the umbrella-decision machine for real, with actual numbers instead of clouds and forecasts. A neuron does exactly three things, in order, and each one is worth a paragraph.

Step 1, multiply.Each input gets scaled by its own weight. A weight of +2.00 means "this clue counts double". A weight of +0.10 means "I barely care about this". A weight of -1.00 means "more of this clue pushes the answer the otherway". Sign is the important part: blue for positive, orange for negative, as on every diagram here.

Step 2, add. Sum all the scaled inputs, then add the bias. The bias belongs to the neuron, not to any input, and it does not care what the inputs are. It shifts the whole answer up or down by a fixed amount. The result of this step is called the pre-activation, and it is written z. You will see z on every page from here to the end of the module.

Step 3, activate. Feed zthrough one fixed curve and out comes the neuron's activation, written a. This step has no weights and nothing to learn, with one exception (PReLU) covered two pages from now. It is the same curve for every neuron in a layer, applied to one number at a time.

That is a neuron. Everything else in this module is many of these, arranged, and a procedure for choosing the weights.

Here is one neuron, fully worked, with numbers that do not move. Two clues about the weather, both already scaled to a 0-to-1 range. This is the first hidden neuron of the small network that the rest of this module reuses on every page, so it is worth getting comfortable with.

One symbol appears below that has not been introduced, and it will appear in almost every formula from here to the end of the module, so here it is once. e is a fixed number, 2.718282 to six places, in the same way that pi is a fixed number. It turns up whenever something grows or shrinks smoothly, which is why it ends up inside every curve on this site. e^-1.400000 means e raised to the power -1.4, and it works out to 0.246597. You do not need to know where e comes from to follow anything here. Take it off a calculator and carry on.

Read that last line as "the neuron is 80.2% convinced". The three steps never change. Only the numbers do.

One thing to expect, so it does not look like a mistake later. Sigmoid was chosen here because 0.802184 reads straight off as "80.2% convinced", which is the clearest possible thing for a first worked example. From the next page onward this same neuron uses tanh instead, and its answer changes. That is not a correction to this page; it is a better choice for a neuron sitting inside a network rather than at the end of one, and page 4 explains exactly why sigmoid is the wrong curve for a hidden layer. The neuron, the inputs and the weights stay identical throughout.

This exact configuration is reproducible in Figure 02 above. Set x1 = 0.80, w1 = 1.50, x2 = 0.60, w2 = 2.00, bias = -1.00, activation Sigmoid. The widget calls these weights w1 and w2 rather than w11 and w21 because it only has the one neuron, so there is no second neuron to tell them apart from: its w1 is this page's w11, and its w2 is this page's w21. The widget's readout should show the same 1.40 and the same output. If it does not, the page is wrong and should be reported.

What the weight actually does

Change one weight and nothing else, and watch where the total lands.

The forecast input never moved. It sat at 0.60 the whole time. All that changed is how much this neuron cares about it, and the answer swung from 80% convinced to 40% convinced, crossing from "yes" to "no" on the way. That is the entire meaning of a weight: not what the evidence is, but how much this particular neuron trusts it.

At w21 = -1.00, drawn orange, the forecast is now evidence against rain. That is obviously wrong for weather, and it is exactly the kind of wrongness training exists to fix. An untrained network's weights are random, so roughly half of them start out backwards like this.

What the bias actually does

The bias is the one number that ignores the inputs entirely. Hold everything else fixed and move only the bias.

Same evidence, three different verdicts, ranging from 91.7% convinced to under half convinced. The bias is how easily this neuron is persuaded. A large negative bias makes it a sceptic that needs a lot of evidence before it says yes; a positive bias makes it credulous.

A useful way to see it: the bias is just a weight on an input that is always 1. On the previous page the same decision used three clues, the third being "got soaked yesterday", pinned at 1.00 with a weight of +0.30 and a bias of -1.30. Because that third input never varies, its contribution is a constant 0.30, and folding it into the bias gives -1.30 + 0.30 = -1.00, the exact bias used here. Same neuron, same answer of 1.40, one fewer moving part. Some textbooks write every network this way, with the bias hidden as a permanently-on input. This site keeps it separate because it is easier to read.

Here is the geometric fact that motivates the whole next page, made concrete rather than asserted.

First, one link back to the number you just computed. The neuron's answer was 0.802184, a probability, and a probability above 0.5 is the neuron leaning yes. But sigmoid has a convenient property: it passes through exactly 0.5 when its input is exactly 0, and it only ever climbs, so a1 is above 0.5 exactly when z1 is above zero. "The probability came out above half" and "the total came out above zero" are the same statement about the same neuron, checked on two different numbers. That means you can look for the yes/no boundary in the easier of the two, z1, and read the answer off the harder one for free.

So the neuron says yes when z1 is above zero and no when it is below, and the boundary between yes and no is the set of inputs where z1 is exactly zero:

x2 = 0.50 - 0.75 * x1is a straight line, and no choice of weights or bias will ever make it anything else. The weights set the line's tilt; the bias slides it back and forth without changing the tilt. A neuron with three inputs draws a flat plane in three dimensions instead, and with more inputs a flat slice of higher-dimensional space, but it is always flat. That flatness is precisely what one neuron cannot escape, and it is why the next page needs more than one.

Now that you have watched it move, here is the same thing in the notation the rest of this module uses. Nothing new is happening; this is shorthand for the arithmetic above.

z is the pre-activation, a is the activation, f is whichever curve you chose. When there are many inputs, writing out the sum gets tedious, so the multiply-and-add step gets a single symbol:

The dot is a dot product, and it means exactly "multiply matching pairs and add up the results", which is Steps 1 and 2 with a shorter name. For our neuron, w = (1.50, 2.00) and x = (0.80, 0.60):

Same number, fewer characters.

One more name, for orientation: a single neuron with a sigmoid activation is exactly logistic regression, a model that predates neural networks by decades and is still the default first thing to try on a binary classification problem. If you already know logistic regression, you already know a neuron. If you don't, you now know logistic regression.

Nothing above depends on there being two inputs. With seven hundred inputs, step 1 is seven hundred multiplications and step 2 is a sum of seven hundred and one numbers including the bias. The count of parameters in one neuron is always "one per input, plus one", so a neuron reading seven hundred inputs owns seven hundred and one numbers and not one thing about the three steps has changed.

One simplification worth naming.Every number on this page was chosen to be readable, and the inputs were handed to you already scaled to a 0-to-1 range. Real data does not arrive like that. A raw temperature in Celsius and a raw price in rupees differ by orders of magnitude, and feeding both into the same neuron unscaled makes one clue's weight matter thousands of times more than the other's before any learning has happened. Rescaling inputs to a comparable range is a routine, mandatory preprocessing step, and the initialization pages explain what breaks when it is skipped.