What Is a Neural Network?

Everything in this module is a variation on the paragraph you just read. Three clues, a number saying how much each one matters, and one combined answer. What changes from here is the level of detail, not the idea.

The name gives the wrong impression, so it is worth spending one sentence on. A "neuron" here is not a model of a brain cell. It is a single arithmetic step: multiply some numbers, add them up, apply one fixed curve to the total. The word is a historical borrowing from a 1943 paper that used brain cells as loose inspiration, and it stuck. Nothing on this site depends on the biology being accurate, and it isn't. Read "neuron" as "one small arithmetic unit" and you will never be misled.

The three layers, named

A network is these small arithmetic units arranged in layers, and there are only three kinds of layer to know.

The input layer is not really made of neurons at all. It is just your clues, sitting there as numbers. It computes nothing. If you have two clues, your input layer is two numbers wide.

The hidden layersare where the actual arithmetic happens. They are called hidden for an unglamorous reason: nobody outside the network ever looks at their values. You hand the network an input and read its answer, and everything in between is internal bookkeeping you never inspect unless you deliberately go looking. A network can have one hidden layer or a hundred. "Deep learning" just means "more than a couple".

The output layerproduces the answer you actually asked for. Its shape is decided entirely by the question. One number for "how many millimetres of rain". One number for "will it rain, yes or no". Three numbers for "dry, drizzle, or downpour".

Data moves in one direction through this, left to right, input to hidden to output. Running one input all the way through and reading the answer is called the forward pass, and it is the entire subject of the next two pages.

Here is the umbrella decision above, done with actual numbers rather than clouds. Three clues, each scaled onto a 0-to-1 range, each with a weight saying how much it counts. Blue weights are positive, meaning "more of this clue pushes toward yes". Orange weights are negative, meaning "more of this clue pushes toward no". That colour convention holds on every diagram on this site and never changes.

Read the three weights before reading the total. The forecast is trusted most, at +2.00. The sky is trusted next, at +1.50. Yesterday's soaking barely counts, at +0.30, which is the network's way of saying "that is weak evidence, but it is not nothing".

The biasof -1.30 is the one number not attached to any clue. It is the network's standing scepticism: start 1.30 in the hole, and let the evidence climb back out. Without it, a day with mild evidence everywhere would still come out positive, because every clue here counts towardrain and no clue's value can drop below zero, so the total could never be anything but positive. You would carry an umbrella every day of your life.

The total came out +1.40, comfortably above zero, so the evidence cleared the scepticism and the answer is "bring it".

A real network does not stop at "above zero or not". It passes that 1.40 through a curve that turns it into something readable, and the standard choice for a yes/no question gives 0.802184, which you can read as an 80.2% chance of rain. Which curve, why that one, and where 0.802184 comes from are the next three pages. The arithmetic above is the part that never changes.

One simplification, flagged rather than buried: the "above zero means yes" step is not what a real network does. Real networks use a smooth curve rather than a hard cutoff, for reasons the gradient pages make unavoidable. The arithmetic is the same either way; the cutoff is the part that gets replaced.

Where the numbers come from

A network only ever sees numbers. Everything you feed it has to be turned into numbers first, and that conversion is a real design decision, not a formality.

A measurement is easy. Temperature, price, age, pixel brightness: use the number, usually rescaled so it lands in a small range like 0 to 1 or -1 to 1. The rescaling matters more than it sounds like it should, and the initialization pages explain why.

A categoryhas no natural number. "Monday" is not 1 more than "Sunday". The standard fix is one-hot encoding: give each category its own input, set that one to 1 and all the others to 0. Seven days becomes seven inputs, exactly one of which is hot on any given day.

An image is already numbers. A 28x28 greyscale image is 784 brightness values, and the simplest thing you can do is line them all up as 784 inputs. That works, badly, and the reason it works badly is the entire motivation for convolutional networks.

Text is the hard case and is out of scope here.

Each individual input number is called a feature. The correct answer you are training toward is called the label or the target. A dataset is a pile of (features, label) pairs.

That diagram is the arithmetic above with names attached to positions. The two blue edges are the multiplications, the middle circle is where the adding finishes and the number there is called z, and the right-hand circle holds what comes out of the curve, which is called a. Those two letters are used for the rest of the module and are worth attaching to a picture once, here, before either is used in a sentence.

Everything in rungs 2 and 3 is this, once, in symbols:

and with this page's own numbers substituted in:

w is a weight, x is an input, b is the bias, z is the total before the curve, a is the answer after it, and f is whichever curve you chose. Five letters. They do not change for the rest of this module, and the next page introduces each of them properly rather than in a list.

The example above had three inputs and one neuron. Nothing about it changes when there are seven hundred inputs and forty thousand neurons except how long the sum is, which is a machine's problem and not yours.

The three shapes of question

Nearly every problem in this module is one of three shapes, and the shape decides what the output layer looks like. It is worth knowing the names now, because the last two pages of this module are entirely about getting this right.

Regression. The answer is a quantity. How many millimetres of rain, what price, how many minutes. One output number, and it is allowed to be any size, positive or negative.

Binary classification. The answer is yes or no. Will it rain. Is this email spam. One output number, squeezed to sit between 0 and 1, read as a probability.

Multi-class classification. The answer is exactly one option out of several. Dry, drizzle, or downpour. Cat, dog, or bird. One output number per option, all of them squeezed so they are positive and add up to exactly 1, read as a probability spread across the options.

There is a fourth shape people trip over: multi-label, where more than one answer can be true at once. A photo can be outdoor and daytime and contain people. That is not multi-class, and using multi-class machinery for it produces confidently wrong answers. The output-activations page takes that apart properly.

What "learning" actually means

The weights above (+1.50, +2.00, +0.30, and the bias -1.30) were hand-picked to make the arithmetic readable. In a real network nobody picks them. They start as small random numbers and get corrected. Collectively they are called the network's parameters, and they are the only thing that changes during training. The structure stays fixed; the numbers move.

The loop has four steps and never gets more complicated than this:

1. Forward pass. Push an input through and get an answer. That is the whole of the next two pages.
2. Loss. Compare that answer to the correct answer and produce one number measuring how wrong it was. Bigger number, more wrong.
3. Backward pass. Work out, for every single parameter, whether nudging it up or down would have made the loss smaller, and how strongly. That answer, a direction and a size, is called the gradient. The size matters as much as the direction and will do real work from page 4 onward: a large gradient means this parameter has a lot of influence on the loss and should move a lot, a gradient of zero means moving it would change nothing at all, so it does not move.
4. Update. Nudge every parameter a small step in the direction that reduces the loss. The piece of machinery that decides how big that step is is called the optimizer, and it gets its own page. Repeat, thousands of times.

One full pass over all your training data is one epoch. Training is that loop; inference is step 1 on its own, once the parameters have stopped moving, which is what happens every time you actually use a trained model.

Nothing else in this module is a new idea. It is that loop, examined one part at a time.