The GPU: thousands of simple workers

The problem

A game screen is millions of tiny sums

Your screen is a grid of dots called pixels. An ordinary HD screen has 1,920 across and 1,080 down: about 2 million pixels. A game redraws every one of them 60 times a second, and working out each pixel’s colour (light, shadow, texture) takes dozens to hundreds of sums.

The striking thing is not the size of the number. It is that every pixel’s colour is worked out with the same recipe, just on different inputs. Pixel 1 and pixel 2 million do not need each other’s answer. That is the shape of work Part 4, chapter 14, called almost perfectly splittable.

The analogy

One professor, or a stadium of students

You have a million simple sums to do. You could hire one brilliant professor who works very fast and handles any problem you throw at them. Or you could fill a stadium with ten thousand students who can each only do simple sums, all following the same instruction read out over a loudspeaker. For a million identical sums, the stadium wins easily.

Mapping it back: the professors are CPU cores, the students are a GPU’s many small arithmetic units, and the loudspeaker is one instruction shared by a whole group. Where the analogy breaks: a GPU’s students are not stupid; each can do any sum. They are just poor at going their own way, which chapter 6 shows.

Try it

Paint a picture with more and more workers

Here is a small picture of 1,296 pixels. Each pixel’s colour comes from the same little formula. On each tick, every worker paints one pixel. Choose how many workers you have and press “Paint”:

With one worker the picture takes 1,296 ticks. With 1,296 workers it takes one. That is the entire idea of a GPU: when every piece of work is independent, more workers means proportionally faster. A big data-centre GPU has about 17,000 of these simple workers public.

The trade

Throw away the cleverness, fill the space with arithmetic

Part 4, chapter 12, showed that most of a CPU core is not arithmetic: it is caches, guessing machinery and reordering machinery, all there so one stream of instructions never waits. A GPU makes the opposite trade. It keeps the arithmetic, shrinks almost everything else, and repeats that tiny design thousands of times.

A GPU worker cannot guess ahead, run instructions out of order or keep much in a cache of its own. On its own it would be slow at everyday work. But it is small and cheap, so thousands fit, and for work like painting pixels, that is a far better use of the same silicon.

Marching together

Workers come in groups that share one instruction

Even simple workers each need someone to tell them what to do. Giving every one its own fetch-and-decode machinery (Part 4, chapter 4) would waste the space just saved. So workers are bundled into groups of 32 that all receive the same instruction at the same moment and do it on their own numbers, like a marching band all taking the same step.

NVIDIA calls such a group a warp, after the threads on a loom public. Part 4’s wide CPU instructions (chapter 15) were the same idea in a small dose: one instruction, many numbers. A GPU is built around it from top to bottom.

The gotcha

When workers disagree, they take turns

What happens if a program says “if this pixel is in shadow, do extra work”? Some workers in a group need the extra steps and some do not. But they share one instruction. So the group runs the extra steps for the ones that need them while the others stand idle, then everyone carries on together.

This is called divergence. With shadows scattered at random, even 5% of shadow pixels means most groups contain at least one, so most groups pay for the extra steps. Bunch the same pixels together and only a few groups pay. GPU programmers spend a lot of effort arranging work so that neighbours take the same path. AI maths, happily, has almost no “if” at all.

Hiding the wait

No time to wait? Switch to another group.

A GPU worker has to fetch numbers from memory just like a CPU, and it waits hundreds of ticks too (Part 4, chapter 7). It cannot guess ahead or reorder. Its trick is simpler: keep many groups ready at once. When one group is waiting for memory, switch to another that is ready, instantly, on the next tick.

Switching costs nothing, because each group keeps its own registers on the chip the whole time. A data-centre GPU like NVIDIA’s H100 can keep up to 2,048 workers ready per block of the chip, and over 270,000 across the whole chip public. CPUs spend switches on not waiting. GPUs spend registers on having something else to do.

Feeding the stadium

A motorway, not a sports car

Ten thousand workers eat numbers very quickly, so a GPU’s memory is built differently. A CPU’s memory is tuned for getting one number back soon. A GPU’s is tuned for moving enormous amounts at once, even if each trip is no faster. Think of a sports car versus a 32-lane motorway: the car is quicker for one trip, the motorway moves far more traffic per hour.

The amount of data moved per second is called bandwidth. Data-centre GPUs get theirs from HBM, “high-bandwidth memory”: memory chips stacked on top of each other and placed right next to the GPU in the same package public. Part 8 is entirely about why, for AI, this bandwidth often matters more than the arithmetic.

A lucky accident

Built for games, adopted by AI

GPUs were not designed for AI. For two decades they were built to draw games, and they got faster every year because gamers kept buying them. Then researchers noticed that the sums inside a neural network look a lot like the sums for drawing pixels.

In 2007 NVIDIA released CUDA, a way to program its graphics cards for any maths, not just pictures public. In 2012 a neural network called AlexNet, trained on two ordinary gaming GPUs, won a famous image-recognition contest by a wide margin public. That result kicked off the modern AI boom, and GPUs have been its engine ever since.

Why AI fits

AI is mostly one shape of sum: the grid multiply

Almost all of a language model’s work is one operation: multiplying two grids of numbers, called a matrix multiply. Each number in the answer is made by taking one row of the first grid and one column of the second, multiplying them pair by pair and adding up: the same multiply-and-add score as Part 1, chapter 14. Tap any cell of the answer:

Look at what that means. Every cell of the answer needs only its own row and column, so all 16 cells can be worked out at the same time by 16 workers. In a real model the grids have thousands of rows and columns, giving millions of independent cells. No “if”, no waiting on each other: the perfect stadium job.

Built for grids

Tensor cores: a whole block per step

Once AI became the main customer, GPU makers added units built only for grid multiplies. Instead of one multiply-and-add per worker per tick, one of these units multiplies two small blocks of numbers and adds the result in a single step. NVIDIA calls them tensor cores, and its first ones, in 2017, did a 4-by-4 block per step: 64 multiply-and-adds at once public.

The gain is huge. NVIDIA’s H100 does about 67 trillion sums per second with its ordinary workers, but about 990 trillion with its tensor cores, roughly 15 times more public. On an AI GPU today, the “thousands of simple workers” mostly handle the odd jobs, and the tensor cores do the heavy lifting.

Smaller numbers

Fewer bits, more sums

A number stored with 16 bits takes a multiplier about four times the switches of an 8-bit one, since every bit of one number meets every bit of the other (Part 3, chapter 8). So the same chip can do more small-number sums than big-number ones. AI models turn out to work well with surprisingly rough numbers, so newer GPUs offer smaller and smaller formats:

Smaller numbers also take less memory and less bandwidth: an 8-billion-number model is 16 GB at 16 bits, 8 GB at 8 bits and 4 GB at 4 bits. The cost is rounding error, which the bonus study, Inside LLM serving, looks at closely.

Head to head

How many tokens a second, and what limits it?

Time to compare. For one person chatting with an 8-billion-number model at 16 bits, a chip has to do about 16 billion sums per token (Part 3, chapter 18) and read all 16 GB of numbers from memory for every token (chapter 12). Whichever takes longer sets the speed. Pick a chip:

Two surprises. A data-centre GPU does about 2,600 times more sums per second than a laptop CPU. And for a single person, arithmetic is not the limit at all: the chip could do the sums for tens of thousands of tokens a second, but can only read the model from memory a few hundred times. The fix, serving many people from one read, is the heart of the bonus study, and memory itself is Part 8.

The other catches

What a GPU is bad at

None of this makes the GPU better in general. It is a specialist, and it gives things up:

Working together

The CPU hands work over a narrow bridge

In most computers the GPU is a separate card with its own memory. The CPU prepares the data, sends it across a connection called PCIe, tells the GPU what to run, and collects the result. That bridge is much slower than the GPU’s own memory, so good programs send data across once and keep it on the GPU.

Laptops and phones often take a different route: the GPU sits on the same chip as the CPU and shares one pool of memory. Apple’s M-series chips work this way public. There is no bridge to cross, but the shared memory is slower than a big GPU’s own.

Family tree

Your laptop’s GPU and a data-centre GPU

They are the same kind of chip, at very different scales:

Phones and new laptops also carry a smaller relative called an NPU, a neural processing unit: a tiny block built only for AI grid multiplies, sipping very little power. Part 10 compares the whole family, including Google’s TPUs and Amazon’s Trainium, which are not GPUs at all.

The software moat

Why it is mostly one company’s GPUs

A chip is only useful if programs can use it. CUDA, NVIDIA’s programming system, has been around since 2007, and almost every AI tool was written for it first. Other companies make capable chips, AMD among them, but software built for CUDA has to be adapted to run on them, and that takes time and effort public.

This is why people say NVIDIA’s real advantage is software as much as silicon. It is also why the biggest AI companies design or buy other chips too: Anthropic, for example, has said it uses Google TPUs and Amazon Trainium alongside NVIDIA GPUs public.

Back to AI

What a GPU does while writing one token

Put Parts 2 to 5 together. Each new token (Part 2) means one full pass through the model. That pass is a long series of grid multiplies, one after another, layer by layer. Within each one, the GPU spreads millions of independent multiply-and-adds across its tensor cores, keeping many groups ready so nobody waits on memory for long. Then the next grid, and the next. At the end out comes one list of scores, and one token.

Two limits remain. The layers must happen in order, so even a perfect GPU takes some time per token. And the numbers have to come from memory every time. Those two limits are why the chip track continues: Part 8 is memory, Part 10 is chips designed only for AI, and Part 12 is how thousands of GPUs are wired together to train a model.

The whole picture

Thousands of simple workers, marching in step

A GPU takes the switches a CPU spends on cleverness and spends them on arithmetic instead. Its workers march in groups of 32, hide waits by switching groups, eat data from very wide memory, and, on AI chips, hand the heaviest work to tensor cores that do whole blocks at once. It is poor at chains of decisions and brilliant at piles of identical sums. AI, it turned out, is almost nothing but piles of identical sums.