Learning is nudging numbers

The problem

Billions of numbers, and nobody wrote them

Part 1 showed each token’s list of numbers, and Part 2 showed the model turning numbers into scores for the next word. A large model has billions of these learned numbers. They are called weights or parameters. No engineer chose any of them. If you filled them with random values, the model would write random gibberish. Training is how random numbers become a model that writes.

The analogy

Getting a shower to the right temperature

You step into a shower that is too cold. You do not calculate anything. You feel how wrong it is, turn the tap a little towards hot, feel again, and repeat. If it overshoots and gets too hot, you turn back a little. After a few nudges it is right. Training is exactly this loop, done by arithmetic.

Mapping it back: the tap is one learned number, the water temperature is the model’s answer, “too cold by this much” is the error, and the nudge is a small change to the number. Where the analogy breaks: a shower has one or two taps. A language model has billions, all affecting each other, and you get to nudge all of them at once. The rest of this part is about how that is possible.

The smallest model

A model with one knob

Start tiny. You have the distance and fare of ten taxi rides, and you want a model that guesses the fare from the distance. The simplest model is one number, the price per kilometre: fare = price × distance. That price is the model’s only knob. Turn it and watch how wrong the guesses are:

Too low, and every guess falls short. Too high, and every guess overshoots. Somewhere in between, the misses are as small as they can be. Finding that spot by hand is easy with one knob. The question is how to find it with a rule a computer can follow, because that same rule will then work for a billion knobs.

Measuring wrongness

One number that says how wrong the model is

To improve, the model needs one number that says how bad it is right now. Take each ride’s miss (guess minus real fare), square it, and average them all. That is the loss. Part 2, chapter 19, used a different recipe for grading next-word guesses, but the job is the same: one number, lower is better.

The landscape

Every knob setting has a height: the loss

Try every possible price and plot the loss for each. You get a valley. Each position across is a setting of the knob; the height is how wrong that setting is. Training means walking to the bottom of the valley. The trouble is that the model cannot see the whole valley: working out the loss everywhere is far too expensive for a real model. It can only feel the ground where it is standing.

Feeling the ground means one thing: which way does it slope, and how steeply? That is the slope, drawn as the purple line. Here is the key fact of all machine learning: the slope can be worked out exactly, with arithmetic, from where you stand. And it points uphill. So you step the other way.

Walking downhill

Step against the slope, again and again

The rule is short: work out the slope, take a step downhill whose size is the slope times a small number called the learning rate, and repeat. Steep ground gives a big step; nearly flat ground near the bottom gives a tiny one. Start at a bad guess and watch the steps. Then try different learning rates:

A tiny learning rate creeps and wastes time. A good one gets close in a handful of steps. Too big, and each step jumps past the bottom and lands higher up the other side; far too big, and it climbs out of the valley altogether. This walk is called gradient descent (gradient is another word for slope), and picking the learning rate is one of the most important choices in training any model.

Two knobs

Two knobs make a bowl

Real taxi fares also have a fixed starting charge. Now the model has two knobs: fare = start charge + price × distance. Every pair of settings has a loss, so the landscape is a bowl seen from above, with darker meaning lower. The slope now has two parts, one per knob, and together they point straight uphill. Click anywhere to start there, then walk:

Notice the valley is a long, narrow trench, not a round bowl. The first step drops into the trench, then the walk creeps along its nearly flat floor. Raise the learning rate to hurry it and the steps start bouncing between the trench walls instead. Real loss landscapes are full of trenches like this, which is why training uses cleverer step rules than the plain one here (see Go deeper).

Scaling up

A billion knobs: same walk, no pictures

One knob made a curve. Two made a bowl. A model with 8 billion knobs makes a landscape with 8 billion directions, which nobody can picture. But the rule does not care. Work out the slope for every knob, nudge every knob a little against its own slope, and repeat. Imagine a hiker in thick fog on a mountain range with a billion directions, who can only feel the ground under their feet. They keep stepping downhill, millions of times.

Two things are surprising. First, that this works at all in so many directions, rather than getting stuck in some small dip. With billions of directions there is nearly always some direction that still leads down. Second, that the slope for every one of billions of knobs can be worked out at once, for only about twice the cost of the guess itself. That is the next chapter.

Sharing the blame

Work backwards to find each knob’s share of the blame

A model is a chain: the input goes through one step, then another, then another, each with its own knobs, and out comes a guess. When the guess is wrong, each knob deserves some share of the blame. The trick is to work backwards from the error. The last step’s knobs get blamed first. Then that blame is passed back to the step before, scaled by how much that step affected the next, and so on, all the way to the start.

Think of a relay team that lost a race: you look at the final runner first, then pass the question back down the line, asking each runner how much they changed the result. This backwards pass is called backpropagation, published in its modern form in 1986 public. It hands every knob its exact slope, all in one sweep.

The price of learning

Learning costs three times as much as guessing

Part 3, chapter 18, showed that one guess (one token) costs about 2 sums per learned number. Training needs the guess, then the backwards pass, which costs about twice as much again. So training on one token costs about 6 sums per learned number public.

Batches

Don’t read the whole library before each step

The true slope would use every example in the training data. For a language model that is trillions of tokens, so one step would take months. Instead, each step looks at a random handful, called a batch, and uses its slope as an estimate of the real one. The estimate is a bit noisy, but steps become millions of times cheaper, so the walk wobbles its way down far faster.

Language models typically use batches of a few million tokens per step public. The wobble even helps a little: it shakes the walk out of small, shallow dips that a perfectly smooth walk could settle into.

The language version

Every word in every sentence is a quiz

For a language model, the guess is Part 2’s next-token guess, and the loss is Part 2’s penalty: how small a chance the model gave the token that really came next. The beauty is that any text at all is training data, with no labelling needed. Every position in a sentence is a free quiz: hide the next token, let the model guess, check the answer.

One sentence of seven tokens is six quizzes. A trillion tokens of books, websites and code is about a trillion quizzes, and after each batch every knob is nudged to do a little better next time. Nobody teaches the model grammar, facts or reasoning. Whatever helps it pass the quizzes is what its numbers drift towards.

Try it

Train a 729-knob model, right here

This is a real language model, just a very small one. It guesses the next letter from the current letter only, using a grid of 27 × 27 = 729 knobs (26 letters plus space). It starts with random knobs and learns from this page’s own text, a few thousand letters, using exactly the walk from chapter 6: compute the loss, find every knob’s slope, nudge, repeat.

At first it writes noise, and after “t” every letter is about equally likely. After a hundred or so steps it has learned, on its own, that “h” often follows “t”, that “q” is followed by “u”, and roughly how long words are. It cannot learn more than that, because it only sees one letter back. A real model sees thousands of tokens back and has billions of knobs, but the walk is the same.

Learning versus memorising

Getting the quiz right is not the same as understanding

A model with enough knobs can memorise its training data: give perfect answers on text it has seen, and poor ones on anything new. That is called overfitting, like a student who memorises last year’s exam answers. To catch it, some text is held back and never trained on. If the loss on that held-back text stops falling while the training loss keeps dropping, the model has started memorising.

Large language models are trained on so much text that they see most of it only about once, so classic overfitting is less of a problem for them. But they still memorise passages that appear many times, such as famous quotes or common code, which is a known issue public.

The gotcha

It does not learn while you chat

Because a chatbot remembers what you said earlier in the conversation, it is natural to think it is learning from you. It is not. Once training ends, the numbers are frozen. Your chat only adds text to what the model reads (Part 2, chapter 15). Close the chat and that text is gone from the model’s view; its numbers were never touched.

This is also why a model does not know about events after its training data was collected, unless a tool like web search hands it fresh text to read (part 16).

How big

The bill for training a large model

Six sums per number per token, times the number of knobs, times the number of tokens read. Pick a model and see what it takes:

For a real check, Meta trained Llama 3’s 405-billion-number model on about 15 trillion tokens using up to 16,384 H100 GPUs, and reported about 38 trillion trillion sums in total public, matching the rule above. That is why training frontier models costs tens to hundreds of millions of dollars, and why it happens rarely, while using them happens billions of times a day.

Many GPUs

Thousands of GPUs, one walk downhill

No single GPU could do this in a lifetime, so training is split. The simplest split: give each GPU its own share of the batch, let each work out the slopes from its share, then average all the slopes together so every GPU takes the same step. The model stays identical everywhere; only the reading is shared out.

Averaging means every GPU must talk to every other GPU after every step, which is why AI data centres need very fast connections between chips. For the largest models even one copy is too big for one GPU, so the model itself is split up as well. That is part 12, “Many chips as one”.

The whole picture

Guess, grade, share the blame, nudge, repeat

Training is a walk downhill in fog. The model guesses, the loss says how wrong it was, backpropagation hands every number its share of the blame, and each number takes a tiny step to be a little less wrong. Do that millions of times over trillions of quizzes and random numbers become a model that writes. What exactly those numbers compute, inside, is the subject of part 7 (attention) and part 9 (the transformer).