The only job: guess the next word

The whole job

Guess the next piece. Add it. Repeat.

Give the model “The cat sat on the” and it does not write a sentence. It produces one guess for the single next token, say “ mat”. That token is stuck onto the end of the text, and the whole, slightly longer text goes back in for the next guess. Round and round, one token per lap, until the answer is finished.

That sounds far too simple to write an essay. The rest of this part is about why it is not: what goes into one guess, how the guess is turned into a choice, and what has to be going on inside for the guesses to be good.

The analogy

Your phone’s keyboard, scaled up enormously

You already use a next-word guesser. When you type “see you”, your phone’s keyboard offers three suggestions, like “tomorrow”, “soon” and “later”. It learned them from what people usually type next. Tap the middle one over and over and it writes a whole sentence, usually a silly one.

Mapping it back: the suggestions are the model’s guesses; tapping one is picking a token; tapping again and again is the loop from chapter 1. Where the analogy breaks: a keyboard looks only at the last word or two, so it loses the thread almost at once. A large model keeps the whole conversation in view, which is the difference between a silly sentence and a useful answer. Chapter 10 shows why that matters so much.

What comes out

A score for every token in the list

At the end of its calculation the model does not output a word. It outputs a score for every single token in its list: one number each, around 100,000 numbers in all. A high score means “this fits well here”. A low or negative score means “this would be strange”. The scores are raw arithmetic results, so they can be any size and do not add up to anything in particular.

Scores into chances

Turn the scores into chances that add up to 100%

Scores are hard to use directly: some are negative, and they add up to nothing in particular. So they go through a fixed recipe that turns them into chances. First, every score is made positive by raising a special number, about 2.718, to the power of the score. That also stretches the gaps: a score 1 higher becomes about 2.7 times bigger. Then everything is divided by the total, so the results add up to 100%.

Slide the score of “moon” and watch what happens to every chance, not just its own:

Because everything is divided by one shared total, the chances compete. When “moon” rises, every other word loses a little, even though its own score never changed. This recipe is called softmax. It is used in this exact form by essentially every language model public.

Making a choice

Pick by rolling a weighted die

Now there is a chance for every token. The usual way to choose is to roll a weighted die: a token with a 42% chance comes up about 42 times in 100 rolls, one with 2% about twice. Most of the time the favourite wins, but not always. Roll it yourself:

The outline shows each token’s chance; the filled bar shows how often it has actually come up. After a few hundred picks the two match closely. This kind of picking is called sampling: each new token is a sample drawn from the chances.

The obvious shortcut

Why not always take the favourite?

It seems wasteful to roll a die. Why not always take the token with the highest chance? That is a real option, called greedy picking, and it gives the same answer every time. But for writing, it tends to go wrong in a particular way: it gets stuck in loops.

Each step on its own is the most likely one, but a whole text made of only the most likely steps is not likely-looking text at all. Real human writing contains plenty of less-likely words. A 2019 study named this problem and showed that rolling the die, with the long tail trimmed (chapter 8), produces far more natural text public. Greedy picking is still useful when you want the same answer every time, for example when extracting a date from a document.

The dial

Temperature: how adventurous the die is

Before the die is rolled, the scores can be stretched or squashed. Divide every score by a number called the temperature. A low temperature makes the gaps bigger, so the favourite wins almost every time. A high temperature shrinks the gaps, so unlikely tokens get a real chance. Temperature 0 means greedy: always the favourite.

Apps choose this setting for you. Low suits facts and code, where you want the most likely answer. Higher suits brainstorming and stories, where surprise is the point. Too high and the text falls apart into nonsense, because rare, wrong tokens start winning. Developers can set it themselves: Anthropic’s API accepts 0 to 1, OpenAI’s 0 to 2 public.

Trimming the tail

Throw away the long tail before rolling

Even a tiny chance adds up when there are 100,000 tokens. If each of 99,000 unlikely tokens has a chance of one in a million, together they still come up about one roll in ten, and one bad token can derail a whole answer. So before rolling, servers usually keep only the most likely tokens.

The most common rule keeps the smallest group of top tokens whose chances add up to some share, like 90%, and throws the rest away. Slide the share:

The kept tokens’ chances are scaled back up to add to 100%, then the die is rolled as usual. This rule is called top-p or nucleus sampling, from that 2019 study public. A simpler rule, top-k, keeps just the k best tokens: the famous 2019 GPT-2 sample texts kept the top 40 public.

Try it

A word guesser, trained on this page

When this page opened, your browser read its own text, a few thousand words, and counted which word follows which. That is a real next-word guesser, of the oldest and simplest kind. It cannot understand anything; it only knows what followed these exact words on this page. Pick how many words back it may look, set the temperature, and let it write.

With one word of view, the text wanders: each word fits the one before it and nothing else. With two or three it starts to sound like this page. Watch the “copied” tile, though: chapter 11 is about what happens at four.

Why view matters

The clue is often far back

Try guessing the last word here: “I grew up in a small village in France, and moved to London when I was twenty, so today I speak fluent ___.” The words right before the gap, “speak fluent”, could end in any language. The clue that decides it, “France”, is sixteen words earlier.

Real conversations are full of this: a name mentioned once at the start, a rule set three messages ago, a variable defined at the top of a file. So instead of a few words, modern models see the entire conversation at once, up to hundreds of thousands of tokens. The mechanism that lets every token look back at every earlier token is attention, the subject of part 7. How much a model can see at once, and what it costs, is part 14.

The gotcha

Seeing more is not enough. Copying is easy.

Set the guesser in chapter 9 to four words and press “Write 25 words”. The text suddenly looks fluent, and the “copied” tile jumps. It is not being clever. With four words of view, almost every four-word run on this page was followed by exactly one thing, so it can only recite the page back.

Most of what you will ever type to a chatbot has never been written before in exactly that form, so a table of counts would have nothing to look up. A real model has to turn the text into meaning, as Part 1’s map began to show, and reason from patterns it has learned. That is why it needs billions of learned numbers rather than a table. Real models can still memorise passages they saw many times, which is a known issue public, but that is the exception, not how they work.

The cost of one token

Every new token is a full run of the model

Each lap of the loop is a complete pass through the whole model: every layer, all its billions of numbers, just to produce one token. The laps cannot overlap, because lap 501 needs the token that lap 500 produced. So a 1,000-token answer is 1,000 full passes, one strictly after another.

It is also why long answers cost more than long questions. A question can be read in one big go, because all of its tokens are already known. An answer can only be written lap by lap. The bonus study, Inside LLM serving, is about exactly this cost and how companies cope with it.

No backspace

Once a token is written, it stays

The loop only ever adds to the end. There is no step that goes back and changes an earlier token. If an unlucky roll puts a wrong word in the middle of a sentence, every later guess has to build on top of it, because that wrong word is now part of the text the model reads.

Newer “thinking” models soften this by writing a private draft first: they reason step by step, can notice a mistake and say “wait, that is wrong”, and only then write the answer you see. Even then, the draft itself is written one token at a time with no backspace. That is part 16.

Is it just autocomplete?

Simple job, hard to do well

“It just predicts the next word” is true about the job, and misleading about what the job takes. Think about what it would take to guess the next word well in every kind of text. In a maths textbook, the next token after “so x equals” is the answer to the problem. In a detective novel, the next word after “the murderer was” needs the whole plot. In code, the next line has to actually work.

So the simple job puts enormous pressure on what happens inside. And the model does more than look one token ahead. In 2025 Anthropic published research that looked inside a Claude model writing rhyming poetry: before writing a line, it had already picked the rhyming word it was heading for, and wrote the line to land on it public. The output is one token at a time; the calculation behind each token can plan further.

Where the assistant comes from

A chat is a document the model continues

If the model only continues text, how does it hold a conversation? The app turns the chat into one long document, with markers for who is speaking, and leaves it open at the start of the assistant’s turn. After a question, the most likely continuation of an open assistant turn is an answer to it. So the model “answers” by continuing the document.

[system] You are a helpful assistant. Be concise.
[user] What is a token?
[assistant] A token is a small piece of text, such as a word or part of a word.
[user] Why not use whole words?
[assistant]▍

A raw model trained only on internet text would continue this in odd ways: it might write the user’s next question too, or wander off into a forum thread. Making it reliably behave like a helpful assistant, and stop at the end of its turn, takes a second round of training. That is part 13.

Knowing when to stop

Stopping is just another guess

The token list includes special tokens that are not text, such as one meaning “end of my turn”. The model is trained so that, when an answer is complete, that token gets the highest score. When it is picked, the loop stops. So stopping is not a separate decision; it is one more next-token guess.

Different every time

Why the same question gets different answers

Ask the same thing twice and you often get two different answers. Now you can see why. The die is rolled at every token, and an early difference sends everything after it down a different path, because each guess builds on what came before.

Across a 500-token answer the number of possible paths is astronomically large, so two identical answers are rare unless the temperature is set to 0. That variety is usually a feature, not a bug: it is why “try again” gives you a fresh attempt.

The honest limitation

Likely is not the same as true

A chance measures how well a token fits text like this, based on everything the model learned. It is not a measure of truth. Usually the two line up, because true statements are common in good text. But where a wrong answer is common, or the model learned too little about something, a false token can get a high chance and be written just as confidently as a true one.

The model has no separate fact-checking step inside this loop. Better training, larger models and tools like web search (part 16) all reduce the problem, and part 15 is entirely about why models make things up and how that is fought.

How a guess is graded

Good guessers give the real next word a high chance

How do you measure whether a guesser is any good? Take some real text, hide each next token, and look at the chance the model gave to the token that really came next. A high chance is good. A tiny chance means the model was badly surprised.

The penalty counts how many times you would have to halve 100% to reach the chance it gave, so it stays small for good guesses and grows fast for bad ones. Averaged over billions of tokens, it is called the loss, and it is the one number training tries to push down. Part 6 shows how: every learned number gets nudged in whichever direction makes this penalty a little smaller.

The whole loop

One token, start to finish

Everything in this part, in the order it happens, every single time a token is written:

Steps 3 to 5 take a tiny share of the time. Nearly all the work is step 2, the model’s huge calculation, and the rest of the series is about what happens inside it: how it learned its numbers (part 6), how words look at each other (part 7), and how the layers fit together (part 9).