How a Neural Network Learns

When your phone recognizes your face or a chatbot answers a question, a neural network is turning numbers into an answer. Nobody wrote down the rules it follows. It started out as a pile of random numbers that produced nonsense, and one simple procedure, repeated millions of times, nudged those numbers until the nonsense became useful. This page builds that procedure from the ground up, starting with a problem small enough to solve by hand.

Here are twelve bike rides. Each dot is one ride: how far it went, and how many minutes it took. A straight line through the dots would let you predict the time of the next ride from its distance. Drag the two blue handles to put the line where you think it fits best:

Every line misses most rides by a little. The red segments show each miss: how many minutes the line's prediction is off for that ride. To turn twelve misses into one number, square each one and take the average. Each red square has a miss as its side, so its area is that miss squared, and the number under the figure is the average area. This is the mean squared error, and it's the first loss on this page: one number that says how wrong a model is.

Squaring does two jobs. It makes every miss count, whether the line runs above the dot or below it. And it punishes big misses far more than small ones: one ride missed by 6 minutes costs as much as nine rides missed by 2. Drag the line far from the dots and the squares balloon. The best line I could find for these rides, the best possible line, still misses by about a minute and a half on a typical ride, an average squared miss of 2.4 min². No straight line does better, because real rides don't fall on a straight line.Fitting a line by making the sum of squared misses as small as possible is the method of least squares. Legendre published it in 1805 for fitting the orbits of comets, and Gauss claimed he had used it since 1795.

One number for how wrong

A straight line is set by two numbers. The slope \blue{w} says how many extra minutes each kilometer adds. The start \purple{b} is the time a ride would take if it went nowhere: unlocking the bike, waiting at the first light. The line predicts that a ride of x km takes

minutes, and the loss averages the squared misses over all twelve rides (x_i, y_i):

Read it as a machine with two dials. Set the dials, and out comes one number. "Learning" will mean turning the dials until that number is as small as it can get. Hold the start at its best value and turn only the slope, from , and watch where the loss goes:

The loss traces a curve with a single lowest point, at a slope of about 3.45 minutes per kilometer. Turn the slope too low and the line runs under the long rides; too high and it runs over them. Either way the squares grow, and because they're squares, they grow faster the further you go: double the distance from the best slope and the extra loss quadruples. That shape, a bowl, is what makes this problem easy. From anywhere on the curve, "downhill" points toward the answer.

A landscape of mistakes

The line has two dials, though, so the loss isn't a curve. It's a surface over a map: every point on the map is one pair of a slope and a start, and its height is the loss of that line. Below, the map is shaded by the loss, darker where it's larger, with contour lines joining points of equal loss, like the height lines on a hiking map. Drag the blue point to choose a line, and see it on the rides at the side:

The lowest point sits at the bottom of a long, narrow valley that runs diagonally across the map. The valley's shape has a reason you can feel in the rides. Raise the start and lower the slope together, like this, and the line pivots around the middle of the dots: it gets worse at both ends, but only a little. Change the slope alone, like this, and every long ride misses by a lot. So the loss rises steeply across the valley and gently along it.

The same landscape in three dimensions makes the valley easier to see. Turn it by dragging, and move the ball with the pad below:

For two dials you could find the bottom by brute force: work out the loss at every point on a fine grid and take the smallest. For this problem there's even a formula that lands on the answer directly. Neither trick survives contact with a real network. A small image classifier has millions of dials, and a grid with just ten settings per dial would need more points than there are atoms in the universe. Nobody can see the landscape. The question is how to find the bottom of a valley you can't see.

Downhill, one step at a time

Standing somewhere on the landscape in the dark, you can't see the bottom, but you can feel which way the ground tilts under your feet. For the loss, that tilt is a number you can compute exactly, right where you stand, without knowing anything about the rest of the map: the slope. Then take a small step in the direction that goes down, and repeat. That's the whole method. It's called gradient descent, and it goes back to Augustin-Louis Cauchy in 1847 .

Back to one dial first. The orange line touches the loss curve at the current slope; how steeply it tilts is the slope of the loss there. The arrow underneath is the step: the tilt times a small number called the step size, in the downhill direction. Press Take a step a few times, or drag the point somewhere else and start again:

Far from the bottom the curve is steep, so the steps are big; near the bottom it flattens, so the steps shrink on their own, and the point settles instead of overshooting. Nobody tells it where the bottom is. It only ever looks at the slope where it stands.

The step size matters a great deal. Set it small and the point creeps. Set it to exactly 0.0147 and one step lands on the bottom from anywhere, because for this curve the slope grows in proportion to the distance from the bottom, at a rate of about 68 per min/km; one over that rate is 0.0147. Go past twice that, 0.031, and each step overshoots the bottom by more than it started, so the point bounces higher and higher up the walls. In a real network nobody knows that rate in advance, which is why the step size, usually called the learning rate, is the first setting people tune.

With two dials the slope becomes a direction, the gradient: how fast the loss changes as you turn each dial, written \orange{\nabla \red{L}} = \left(\frac{\partial \red{L}}{\partial \blue{w}}, \frac{\partial \red{L}}{\partial \purple{b}}\right). Each step moves both dials at once, each by its own slope times the step size. Drag the start point anywhere and watch the path:

The path is lopsided in a revealing way. The first few steps race down the steep side of the valley and the loss collapses, from 29 to under 10 in fifty steps. Then progress slows to a crawl along the floor: it takes another 150 steps to get from 9 to 3. The step size that is safe across the valley is far too timid along it. Try the edge of stability and the path zigzags from wall to wall while it creeps along; one notch more and the zigzags grow until the path leaves the map. Valleys like this, steep one way and nearly flat another, are the normal case in real networks, and much of the craft of training is about crossing them faster. We'll come back to that.

A few examples at a time

Computing the exact slope means going through every example: here, all twelve rides. A real training set might have a billion examples, and a step that waits for all of them is a step that comes once a week. So in practice each step uses a small random handful, a mini-batch, and treats the slope of their loss as an estimate of the real one. Set how many rides each step looks at with , and compare the mini-batch path with the full path that uses all twelve:

With one ride per step, each step is a guess based on a single ride, and some rides pull the line in odd directions, so the path wanders. It still gets close to the floor, because the guesses are right on average: the long-run pull of the random handfuls is the same as the pull of all twelve rides. With four rides the jitter calms down a lot, and each step costs a third as much as using every ride. That trade, many cheap noisy steps instead of a few exact ones, is how nearly every network is trained. The method has a name to match: stochastic gradient descent, from the Greek for "guessing".A pass through the whole training set in mini-batches is called an epoch. Large models are often trained for only one or a few epochs, because their training sets are so large.

The noise may even help. A path that jitters doesn't settle into the first small dip it finds; it gets shaken out of narrow pits, and many researchers think this steers training toward wide, flat valleys. A model sitting in a wide valley gives similar answers even when its numbers are nudged a little, which seems to go with working well on examples it hasn't seen, though exactly why is still argued about.

From lines to decisions

Fitting a line is the simplest kind of learning, and everything so far carries over unchanged to networks. What changes is the model: the thing with the dials. A neural network is built from one small part repeated thousands of times, the artificial neuron. It takes some numbers in, multiplies each by its own weight, adds them up along with a bias, and passes the sum z through a squashing function, here the sigmoid, which turns any number into one between 0 and 1:

Move the input point in the pad and turn the weights; the bias is :

A single neuron already makes a decision. Its output is above one half exactly when the sum z is positive, and z = 0 is the equation of a straight line through the plane of inputs. Every input on one side of that line gets an output near 1, everything on the other side near 0, with a soft edge in between. The weights set the line's direction and how sharp the edge is; the bias slides it. So a neuron is a line-drawer, the same machine as before, aimed at a different question: not "how many minutes?" but "which side?"This is essentially Frank Rosenblatt's perceptron of 1958 , with a smooth sigmoid in place of his hard on/off step, so that the slope of the loss exists everywhere.

To learn a decision, it needs a loss that says how wrong a probability is. Squared error works badly here. Suppose the right answer is "yes" and the neuron says 0.01. Squaring the miss gives 0.98, not much worse than a timid 0.5 would cost. The usual choice is the cross-entropy: minus the logarithm of the probability given to the right answer. Drag the probability and compare:

Cross-entropy charges 0.69 for a coin flip, 2.3 for a probability of 0.1, and 4.6 for 0.01: confident and wrong is expensive, and the slope there is steep, which is exactly what makes learning fast when the model is badly wrong. It also has a pleasingly honest meaning. It's the number of nats of surprise the model feels at the truth, the same quantity a compression program pays to encode an outcome it predicted to have that probability.

Now put it to work. Below are 160 points of two kinds, green and purple. The neuron's line and its soft edge are shown as shading. Drag the two handles on the line to place it yourself, or press play and let gradient descent on the cross-entropy move it:

Started from a bad guess, the line turns and slides until it runs between the two groups, and the edge sharpens as the neuron grows more confident: the weights keep growing as long as making them bigger makes the right answers more certain. A few points sit on the wrong side; the two groups overlap, so no line gets all of them, and the neuron settles where the cross-entropy, not the count of mistakes, is lowest.

What one neuron can't do

Here's a pattern that looks just as simple. Purple points sit in two opposite corners, green points in the other two. It's called XOR, for "exclusive or": a point is purple when exactly one of its coordinates is positive.

No straight line does better than about three corners out of four. This limit of single-layer networks was pointed out sharply by Marvin Minsky and Seymour Papert in their 1969 book Perceptrons , and it helped cool interest in neural networks for more than a decade.The fix, stacking layers, was already known. What was missing was a good way to train the hidden layers, which is the subject of the section on backpropagation below. The way out is to stack neurons: feed the inputs to a few neurons, called a hidden layer, and feed their outputs to a final neuron.

Two hidden neurons are enough. Each draws its own line; here both look at the sum of the two coordinates, with different thresholds. One fires when the sum is above −0.5, roughly "at least one coordinate is positive". The other fires only when it's above +0.5, "both are positive". The output neuron takes the first minus the second: "at least one, but not both". The cleverer way to see it is that the hidden layer moves the points. Slide from the inputs to what the hidden layer outputs:

At the start, the blue and red lines are where the two hidden neurons switch on, and they cut the plane into three bands: the purple points all sit in the middle band. Slide toward the hidden layer and the whole plane folds. Each point lands at the pair of numbers the two hidden neurons output for it, and in those coordinates the purple points have gathered together, away from the green, where a single straight line separates them. That's what hidden layers do in general: they move the data into new coordinates where the final decision is a simple one. Nobody chooses those coordinates by hand. Training finds them.

A network learns a shape

Now everything at once. The network below has two inputs (a point's coordinates), two hidden layers of neurons each, and two outputs that give the probability of each color. Every connection has a weight, every neuron a bias, and all of them start at small random values. Training runs gradient descent on the cross-entropy, 120 mini-batches of 24 points a second. Pick a pattern and press play:

On the circle, the shading starts as a smear and within a few hundred steps closes into a ring around the purple points. Watch the diagram while it trains: the weights change color and width as the network sorts out which hidden neurons should care about which direction. Drag the probe point from inside the ring to outside and you can watch the network's internal response change, layer by layer.

The size of the network matters. With one neuron per layer, the best it can do is a single soft line, and the loss stalls high. On the spiral, sixteen per layer need a thousand or so steps to wind the boundary around both arms; with three per layer it never gets there. A network's size limits the shapes it can draw, and a bigger one can draw more of them.A network with one hidden layer can approximate any reasonable function as closely as you like, given enough neurons: this is the universal approximation theorem. It says nothing about how many neurons that takes, or whether gradient descent will find the right weights.

Blame, passed backward

Everything above depends on one quantity: the slope of the loss with respect to every weight. For the line there were two, and a formula. The network above has a few hundred, and some of the models behind chatbots have hundreds of billions. Nudging each weight in turn and measuring the change would take one pass through the network per weight, per step. The trick that makes it affordable is called backpropagation, made widely known by David Rumelhart, Geoffrey Hinton and Ronald Williams in 1986 , and it gets every slope for about the cost of one extra pass through the network.The idea had been found before: Seppo Linnainmaa described the general method for computing derivatives this way in 1970, and Paul Werbos proposed using it to train networks in 1974. The 1986 paper showed it learning useful hidden features.

It rests on the chain rule. If a small change in a changes h by 0.3 times as much, and a change in h changes the loss by 2 times as much, then a change in a changes the loss by 0.3 × 2 = 0.6 times as much. Slopes along a chain multiply. Here's the smallest chain with all the parts of a network: an input multiplied by a weight, a squashing function, another weight, and a loss. The blue numbers are the forward pass. Step through the backward pass to see the orange slopes flow from the loss back to each weight; the weights are and :

The backward pass starts at the loss, whose slope with respect to itself is 1. Each stage multiplies by one local slope: how much a box's output moves when its input moves. Squaring gives 2(y - t); multiplying by w_2 gives w_2 back toward h, and h toward w_2; tanh gives 1 - h^2. At the last stage the readout below the figure compares the result with the slow way, nudging w_1 up and down by a ten-thousandth and measuring the change in the loss. They agree to four decimal places, for any weights you choose.

Two features make this scale. Each local slope only needs numbers the forward pass already computed, so nothing is recomputed. And when a value feeds several later boxes, as every neuron in a layer does, the slopes coming back from each are simply added. So one forward pass and one backward pass give the slope for every weight at once, whether there are two of them or two hundred billion.

Faster than straight downhill

Back to the narrow valley. Plain gradient descent wastes most of its effort there: the steep walls force a small step, and along the gentle floor that small step is painfully slow. Two fixes are used almost everywhere.

Momentum, introduced by Boris Polyak in 1964 , treats the point like a heavy ball. Each step adds the current slope to a running velocity that slowly forgets old slopes, so pushes that keep pointing the same way, along the floor, build up speed, while the back-and-forth pushes across the valley cancel out. Adam, from Diederik Kingma and Jimmy Ba in 2015 , goes further: it keeps a running average of how big each dial's slopes have been and divides each dial's step by it, so a dial with steep, noisy slopes takes cautious steps and a dial with tiny slopes takes bold ones. Race them from the same start; drag the start to try other places:

Plain descent zigzags down the wall and then plods along the floor; after 100 steps its loss is still above 4, and it needs about 300 steps to reach the bottom. Momentum overshoots at first, swings across the valley, then picks up speed along it and gets there in about 100. Adam heads more directly for the floor, because it divides each dial's step by the size of that dial's recent slopes, which evens out the steep and gentle directions. Push the step sizes up by 40% and plain descent crosses its stability edge and blows up, while the other two cope.

The comparison isn't a general ranking. On a different landscape the order can change, and each method has its own settings to tune. But Adam and its close relatives are the default way large networks are trained today, mostly because they work reasonably well without much tuning across very different problems.

Memorizing instead of learning

Low loss on the training examples is not the goal. The goal is good answers on examples the network has never seen. Those two can come apart, and the way they come apart is one of the most important things to understand about machine learning.

Below, a network with one input and one output learns a curve from noisy samples, the filled blue dots. The hollow red dots are 200 more samples from the same curve with the same noise, which training never sees: a held-out set. Press play and watch both losses:

With eight points and a large network, the training loss plunges toward zero: the curve threads through every blue dot. But to do that, it bends to fit each dot's noise, and between the dots it swings wildly. The held-out loss tells the story. It drops at first, while the network is learning the real shape, bottoms out after a hundred or so steps, then climbs, even as the training loss keeps falling. The network has started to memorize. The green line marks the noise in the data itself, the lowest held-out loss any model could reach.

Two things push back. More data: with 40 points there's too much to memorize easily, the curve stays close to the truth for much longer, and the held-out loss gets near the noise line. And a smaller model: a small network on eight points can't bend as sharply, so it chases the noise far less. In practice people also stop training when the held-out loss stops improving, and use other tricks to keep the weights small and smooth. Every serious result in machine learning is reported on held-out data for this reason.Very large modern networks complicate this picture. Past the point where they fit the training data exactly, making them even bigger can make held-out error fall again, a pattern called double descent . Why is still an active research question.

Where the numbers start

One more thing has to go right before any of this works: the starting weights. All zeros won't do, because every neuron in a layer would compute the same thing and receive the same slope, so they'd stay identical forever. Random values break the tie, but their size matters, and it matters more the deeper the network.

Each layer multiplies its inputs by its weights and adds them up. If the weights are a bit too small, each layer's output is a bit smaller than its input, and after a dozen layers the signal has faded to almost nothing. Too large, and it grows, until tanh neurons are pinned at ±1 where their slope is zero. Either way the backward pass, which runs through the same weights, fades or blows up too. Below is a network of twelve layers of 48 neurons, fed 64 random inputs. Scale its starting weights by :

At 0.6 times the usual scale, the tanh network's outputs shrink layer after layer until the last few are nearly silent, and the slopes reaching the first layers are a thousand times smaller than those at the last: the early layers would barely learn. This is the vanishing gradient problem, and for years it made deep networks hard to train. Xavier Glorot and Yoshua Bengio worked out in 2010 that a random start keeps signals roughly steady when each weight's spread is about one over the square root of the number of inputs it adds up, the 1× setting here. For ReLU neurons, which pass only positive values and so lose half the signal at each layer, the right scale is √2 times larger, as Kaiming He and colleagues showed in 2015 : compare ReLU at 1× with ReLU at 1.4×.

From lines to language

Everything on this page scales up without changing in kind. The chatbot you might have used was trained by the same loop: show it some examples, measure the loss, compute every slope with a backward pass, nudge every weight a little downhill with an Adam-like optimizer, and repeat. Its examples are snippets of text, and its loss is the cross-entropy of predicting each next word, the same −ln p from the figure earlier, averaged over trillions of words. Its dials number in the billions, its landscape has as many dimensions as it has dials, and its mini-batches are millions of words at a time. Nobody has seen that landscape, any more than anyone has seen the bottom of the valley in the dark. They only ever feel the slope.

What this page leaves out is mostly about the shape of the network rather than how it learns : layers that look at images patch by patch, attention layers that let each word in a sentence look back at the others, ways of keeping training stable at enormous scale. The learning underneath all of them is the loop you've been watching: a loss, a slope, and a small step downhill, repeated until the answers come out right.

References

1. Augustin-Louis Cauchy, “Méthode générale pour la résolution des systèmes d'équations simultanées”, Comptes rendus de l'Académie des sciences 25 (1847), 536–538.
2. Frank Rosenblatt, “The perceptron: a probabilistic model for information storage and organization in the brain”, Psychological Review 65 (1958), 386–408.
3. Marvin Minsky and Seymour Papert, Perceptrons: An Introduction to Computational Geometry (MIT Press, 1969).
4. David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, “Learning representations by back-propagating errors”, Nature 323 (1986), 533–536.
5. Boris T. Polyak, “Some methods of speeding up the convergence of iteration methods”, USSR Computational Mathematics and Mathematical Physics 4 (1964), 1–17.
6. Diederik P. Kingma and Jimmy Ba, “Adam: A method for stochastic optimization”, International Conference on Learning Representations (2015); arXiv:1412.6980.
7. Xavier Glorot and Yoshua Bengio, “Understanding the difficulty of training deep feedforward neural networks”, Proceedings of AISTATS (2010), 249–256.
8. Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, “Delving deep into rectifiers: surpassing human-level performance on ImageNet classification”, Proceedings of ICCV (2015), 1026–1034.
9. Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal, “Reconciling modern machine-learning practice and the classical bias–variance trade-off”, PNAS 116 (2019), 15849–15854.
10. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning (MIT Press, 2016). A thorough textbook covering everything on this page and much more.