Text becomes numbers

The problem

A computer only understands numbers

Everything inside a computer is stored as numbers. A photo is a grid of dots, and each dot is three numbers: how much red, green and blue. A song is a long list of numbers saying how far the speaker should push the air, forty-four thousand times a second. Text is no different. It has to be numbers too.

An AI model is a giant calculator. It takes numbers in, does a huge amount of multiplying and adding, and gives numbers out. So the first question of the whole series is simple: which numbers should stand for words? The answer has two steps, and this part covers both: cut the text into pieces, then give every piece a list of numbers that captures what it means.

First try

Give every letter a number

The obvious idea already exists. Every letter, digit and symbol has an agreed number, set by a worldwide standard called Unicode. Lowercase a is 97, b is 98, c is 99. It covers about 150,000 characters public, from English and Hindi to Chinese and emoji. Type below and see the numbers your computer already uses:

This works, and it is how text is stored on disk. But it is a poor way to feed a model, for two reasons. It is long: a 1,000-word essay is about 5,000 letters, and later parts show the model does real work for every single piece it reads and writes. And the numbers mean nothing: 97 and 98 sit next to each other, yet “a” and “b” have nothing in common. Worse, a model might believe they do.

Second try

Give every word a number

So try the opposite: a big numbered dictionary of whole words. “the” is 1, “cat” is 2, and so on. Now a sentence is short, one number per word. But three new problems appear.

Typos, new slang, product names, a word in another language: none of them are in the list, so the model would see only a blank “unknown” marker and lose the word entirely.

The middle path

Cut text into reusable pieces

The answer every modern model uses sits between letters and words. Keep a list of pieces. Common words get one piece each. Rare words are built from smaller pieces that are reused everywhere, like “un”, “follow” and “ed”. And the list always contains every single byte, so even a string of nonsense can be spelled out piece by piece. Nothing is ever unknown.

Each piece is called a token, and the list of all pieces is the model’s vocabulary. Notice the ␣ marks: the space before a word usually belongs to the piece, so “ cat” and “cat” at the start of a sentence can be different tokens. In English, one token is about four characters, or three quarters of a word, on average public.

Building the list

Glue the most common pair, again and again

Nobody writes the piece list by hand. A program builds it from a huge pile of sample text with one simple rule. Start with single letters. Count every pair of neighbours. Glue the most common pair into a new piece and add it to the list. Repeat, tens of thousands of times.

Here is the whole idea on four words, each with how often it appears in the sample. Slide to run the rule one step at a time; the orange pair is the one about to be glued.

After a few steps, “est” and “low” and “new” exist as pieces, because they kept turning up. Run this on billions of words and the list fills with whole common words, common endings like “ing” and “tion”, and pieces of names. This method is called byte-pair encoding, or BPE public: it began as a way to compress files in 1994 and was adopted for language models in 2016.

Try it

A working tokenizer, trained on this page

When this page opened, your browser ran the glue rule on the text of this page itself: a few thousand words. So the tokenizer below is real, just small and trained on very little. Slide to grow its piece list and watch the same text take fewer tokens. Each token’s ID number is underneath.

At 256 pieces the list holds only bytes, so English text is one token per letter. As pieces are added the count drops fast, then more slowly. Dashed boxes are lone bytes that are only part of a character: this small list never saw Hindi or emoji, so it spells them out byte by byte.

The hand-off

The model only ever sees ID numbers

Once text is cut, each token is replaced by its position in the list: its ID. Your message reaches the model as nothing but a row of whole numbers. The model never sees a letter. When it answers, it produces IDs too, and the app looks each one up in the same list to turn it back into text for your screen.

The gotcha

Why a model can miscount letters

Ask a model how many letter r’s are in “strawberry” and older models often got it wrong. Now you can see one big reason. The model does not receive the letters s-t-r-a-w-b-e-r-r-y. It receives a few token IDs, and an ID is just a number. To count letters it would have to have learned, separately, how each of its tokens is spelled.

The same blind spot explains other old stumbles: spelling a word backwards, finding rhymes, and arithmetic on long numbers, which can be cut into odd chunks like “123” and “4567”. Newer models do much better, because they have learned a lot about spelling and some can work step by step (part 16). But the root cause is the cutting.

Not all text is equal

The same message can cost very different amounts

A piece list is built from its sample text, so it is efficient for whatever that sample had most of. Text it saw rarely gets cut into many small pieces. Here is this page’s tokenizer, trained only on English, cutting the same short greeting in different forms:

Real tokenizers learn from many languages, which narrows the gap but does not close it. A 2023 study found the same text could need up to 15 times more tokens in some languages than in English public. That matters, because people pay per token, wait per token, and hit length limits counted in tokens.

Half done

An ID is a name tag, not a meaning

Cutting text and numbering the pieces solves the first job. But an ID is like a locker number. Locker 902 and locker 903 are neighbours, and what is inside them has nothing to do with each other. In the same way, “cat” might be ID 902 and “kitten” ID 74,311, even though they mean nearly the same thing.

If the model did arithmetic on the IDs themselves, it would believe “cat” (902) is almost the same as whatever happens to be 903. So the second job is to swap each ID for numbers that do carry meaning.

The analogy

A map of meaning

Imagine placing every token on a map, so that tokens used in similar ways sit close together. Animals end up in one corner, foods in another, vehicles in a third. Now a token’s numbers are simply its position on the map: how far across, how far up. “cat” and “kitten” get almost the same numbers, because they sit side by side.

Tap any word to see its three nearest neighbours.

Mapping it back. The map is the model’s table of numbers; a word’s spot is its list of numbers; distance on the map is how different two meanings are. Where the analogy breaks: this map has two directions, across and up. A real model’s map has thousands, which no one can picture. The positions here are hand-placed example values.

Thousands of directions

Each token gets a long list of numbers

Two numbers can only separate a few kinds of meaning. Real models give every token a list of thousands: 4,096 numbers in Llama 3 8B, 12,288 in GPT-3 public. Each number is one direction on the map. The list is called the token’s embedding: the word is “embedded” as a point in a space with thousands of directions.

Below, each bar is one number in a 48-number list example; up is positive, down is negative. Similar words get similar patterns.

No single number cleanly means “is an animal” or “is plural”. Meaning is spread across many of them at once, the way a face is not stored in one pixel. That is one reason models are hard to look inside.

Directions carry meaning

Arithmetic on meaning

Something surprising happens on these maps. The step from “man” to “woman” points in roughly the same direction as the step from “king” to “queen”. So you can do sums with meaning: start at king, subtract man, add woman, and you land near queen. This was shown with an early word map called word2vec in 2013 public.

The honest version. It works for famous examples and fails for many others. Usually the nearest point to the sum is one of the words you started with, so those are ruled out by convention. It is a real tendency, not a law. But it shows that directions, not just positions, carry meaning: one direction roughly means “female version of”, another “capital city of”, another “past tense of”.

Measuring closeness

How a model compares two lists of numbers

Think of each list as an arrow from the centre of the map to the word’s spot. Two arrows pointing the same way mean similar meanings. Arrows at right angles mean unrelated. Opposite arrows mean opposite. Turn one arrow and watch the score:

The way to get that score from two lists is pure arithmetic: multiply the two first numbers together, the two second numbers together, and so on, then add everything up. Matching signs push the total up, clashing signs pull it down. This is called the dot product. Scaled so the result runs from −1 to +1, it is called cosine similarity. The same multiply-and-add appears again in part 7, where it is how each word decides which other words to pay attention to.

Where the map comes from

Nobody draws the map. Training does.

No person decides where “kitten” goes. Every token’s list starts as random numbers, so the map starts as a meaningless scatter. During training (part 6), the model reads enormous amounts of text and keeps being nudged to guess the next word better. Words that appear in similar sentences get nudged towards similar positions, because that helps the guessing. Slide to watch a scatter settle into a map:

This is an illustration example: real training moves thousands of numbers per token at once, over weeks. But the principle is old. In 1957 the linguist J. R. Firth wrote, “You shall know a word by the company it keeps.” public A model learns a word’s meaning only from which words keep turning up around it.

The lookup table

One row of numbers per token

Inside the model, the map is stored as a giant table, the embedding table. It has one row for every token in the vocabulary, and each row is that token’s list of numbers. The very first thing a model does with your message is look up the row for each ID. That is the whole of step one: no thinking, just looking up.

The limitation

One word, many meanings

The table gives each token one fixed row. But “bank” in “the river bank” and “bank” in “bank account” are the same token, so they start from exactly the same numbers. The table cannot tell them apart.

Fixing this is the job of the rest of the model. Layer by layer, each token’s list is adjusted using the tokens around it, so “bank” next to “river” drifts towards water and “bank” next to “account” drifts towards money. How the words share that information is part 7, attention. The table is only the starting point.

One more missing piece

Order matters, and the table ignores it

“Dog bites man” and “man bites dog” contain exactly the same three tokens, so the table hands back exactly the same three lists. Yet the meanings are opposite. So the model also needs to know where each token sits in the row.

Modern models mix position into the calculation, so a token “knows” it is first, second or third. The common method today does it by gently rotating the numbers by an amount that depends on position. That is covered properly in part 7, next to attention, because that is where position is used.

The whole trip

From your message to the model’s first numbers

Put together, everything in this part happens before the model does any real work, in a few thousandths of a second: