The CPU: the chip in your laptop

The problem

One chip, any job

An adder can only add. To make one chip do any job at all, you need something that reads a list of steps and carries them out one by one: “fetch this number, add that one, if the result is zero go back three steps”. Change the list and the same chip does a completely different job. That list is a program, and the part of the chip that follows it is the CPU, short for central processing unit.

This idea, that the program is just data the machine reads like any other numbers, was laid out in 1945 and is still how every computer works public.

The analogy

A chef following a recipe

Picture a chef with a recipe book. The chef reads one step, does it, and moves to the next. A few ingredients are in the chef’s hands. More are on the counter. Most are in the pantry, and some are in a warehouse across town. The chef is incredibly fast with a knife, but every trip to fetch an ingredient takes time.

Mapping it back: the recipe is the program, each step is an instruction, the chef is a core, the hands are registers, the counter and pantry are caches, and the warehouse is the main memory, the RAM. Where the analogy breaks: a real chef could never do three billion steps a second, and never has to wait hundreds of steps for an ingredient. A CPU does both, which is what most of this part is about.

Instructions

A program is a list of numbered steps

A CPU understands only a small, fixed set of very simple instructions: put a number in a register, add two registers, copy a value to memory, jump to another step. Each instruction is stored as numbers, like everything else (Part 3, chapter 3). The first number says which kind of instruction it is; the rest say which registers or values to use.

Programmers rarely write instructions directly. They write in languages like Python or JavaScript, and other programs translate that into millions of these tiny steps. A single click in a web browser can run hundreds of millions of instructions.

Try it

A tiny CPU you can step through

Here is a complete program, seven instructions long, that adds up 5 + 4 + 3 + 2 + 1. Every instruction goes through the same three phases. Fetch: read the instruction the program counter points at. Decode: work out what it means. Execute: do it. Press “Next step” and watch:

The whole job takes 19 instructions and 57 steps, because instructions 2 to 4 repeat five times. A real CPU runs exactly this kind of loop, only with billions of steps per second and a few hundred different kinds of instruction.

Registers

A few boxes right next to the arithmetic

The adders can only work on numbers that are right next to them. Those numbers sit in registers: a small set of storage boxes inside the core, built from the switch loops of Part 3, chapter 10. A modern core has a few dozen of them for ordinary numbers public. Reading a register takes less than one clock tick.

That is fast, but there are very few. Everything else, every web page, photo and program, lives further away, and has to be brought in before the core can touch it. How far away, and how long that takes, is the single most important fact about a CPU.

Decisions

Jumps make loops and choices

Instruction 4 in the tiny CPU is the clever one. It says: if B is not zero yet, set the program counter back to step 2. That one jump turns a straight list into a loop. A jump that depends on a value is also how a program makes a choice: if the password is right go here, otherwise go there.

Loops and choices are what make a CPU able to run any program at all. They are also what makes its job hard: until the value is known, the CPU does not know which instruction comes next. Chapter 10 shows how it copes.

The real problem

Memory is very, very far away

Main memory, the RAM, sits on separate chips a few centimetres from the CPU. Fetching one number from it takes around 80 to 100 billionths of a second public. That sounds instant, but at 3 GHz it is about 300 clock ticks: 300 steps the core could have taken. To feel the gap, pretend one tick lasts one second:

On that scale, one trip to RAM is a five-minute walk for a number the core could add in one second. A core that waited for memory on every step would spend over 99% of its life standing still. Real CPUs avoid that with two tricks: keeping copies close by (chapter 8), and finding other work to do while they wait (chapter 11).

Caches

Keep copies close to the stove

A sensible chef keeps the ingredients used most often on the counter, not in the warehouse. A CPU does the same with caches: small, fast memories built right onto the chip, holding copies of whatever was used recently. When the core needs a number, it checks the nearest, smallest cache first, then the next, and only goes to RAM if none of them has it.

This works because programs are predictable in one way: they use the same few numbers, and numbers stored next to each other, over and over. In everyday programs the first cache already has the right number more than 90% of the time example. Caches are why your laptop feels fast even though its memory is hundreds of ticks away.

An assembly line

Start the next instruction before this one finishes

In the tiny CPU, each instruction did fetch, decode and execute before the next one began. Real cores work like an assembly line instead: while one instruction is being executed, the next is being decoded and the one after that fetched. Each instruction still takes several steps, but a new one finishes on nearly every tick.

This overlap is called pipelining. Real pipelines have ten to twenty stages public, which is how a core can finish billions of instructions a second without each one being done in a single tick.

Guessing

At every fork, guess which way

The assembly line has a problem with jumps. The line must start fetching the next instructions before the jump has even been worked out. So which ones? The CPU guesses, using a small memory of what this jump did the last few times. A loop that ran a thousand times will probably run again.

This is called branch prediction, and modern predictors are remarkably good public. A wrong guess costs about 15 to 20 ticks of thrown-away work public, so a program with jumps that are truly random, which no guesser can predict, can run several times slower than the same work without them.

Never idle

Do something else while you wait

A good chef who is waiting for the oven chops the onions for the next dish. Modern cores do the same. They look a few hundred instructions ahead, find ones that do not depend on the number still being fetched, and do those first. Results are then put back in the right order so the program never notices.

This is called out-of-order execution. Combined with having several adders and other units side by side, a modern core can finish four to eight instructions in a single tick when it finds enough independent work public. Every bit of this cleverness takes switches, though, and that is the gotcha in chapter 12.

The gotcha

Most of a CPU is not doing sums

You might expect a chip that “computes” to be mostly arithmetic. A CPU is the opposite. The adders and multipliers are a small part of each core. Most of the space goes to caches, the guessing machinery, the reordering machinery and the control that keeps it all straight.

That is a deliberate trade. Everyday programs are full of jumps, waits and unpredictable steps, so it pays to spend switches on never standing still. But an AI model is the opposite kind of work: the same multiply-and-add, billions of times, in a completely predictable order. For that, a CPU’s cleverness is mostly wasted space. Part 5 shows the chip that makes the other trade.

Many cores

More chefs in the same kitchen

Part 3 showed that clock speeds stopped rising around 2005 (Part 3, chapter 16). Since then, chips have grown by adding more complete cores, each with its own registers, caches and cleverness, side by side on one chip. Your laptop probably has around 8 to 16 cores. The largest server chips have nearly 200 public.

Each core runs its own list of instructions, so a computer can run several programs truly at the same time, or split one big job into pieces. That last part sounds like an easy win. Chapter 14 shows why it often is not.

The limit

More cores do not always mean faster

If ten cooks share a recipe, chopping can be split between them, but the steps that must happen in order, like waiting for the cake to bake, cannot. However many cooks you hire, the in-order part takes just as long. In 1967 Gene Amdahl turned this into a simple rule public. Set how much of a job can be split, and how many cores share it:

With 90% splittable, no number of cores can make the job more than 10 times faster. That is why everyday programs rarely use many cores well. But some jobs are 99.99% splittable: every pixel of a game frame, or every multiply in an AI model, can be done independently. Slide to 99.99% and 10,000 cores, and you are looking at the reason GPUs exist.

One step further

One instruction, many numbers

CPU makers saw this coming. Modern cores have special wide instructions that do the same sum on a whole row of numbers at once: one instruction adds eight pairs, or sixteen. It is like a chef with a knife that cuts eight carrots in one stroke.

This is called SIMD: single instruction, multiple data. Intel’s widest version works on 512 bits at a time, sixteen ordinary-sized numbers per instruction public. It is the CPU borrowing a GPU’s main idea, and it is why CPUs can run small AI models at all.

Back to AI

How fast is a CPU at AI maths?

Put it together: cores, ticks per second, and how many numbers each instruction handles. Part 3, chapter 18, showed that an 8-billion-number model needs about 16 billion sums per token. How many tokens a second could your laptop’s CPU manage, if arithmetic were the only limit?

This assumes one multiply-and-add per number per tick on every core, a simplification example. In practice a laptop CPU running an 8-billion-number model writes only a handful to a couple of dozen tokens a second, because, as Part 8 will show, feeding the numbers in from memory is slower still. Usable for a small model on your own machine; far too slow to serve millions of people.

What it is for

Brilliant at messy, everyday work

None of this makes the CPU a bad chip. It is the best chip ever built for work that is unpredictable, full of decisions, and has to be done in order.

In an AI data centre

The CPU is the manager, not the worker

Even in a server built for AI, there are CPUs. They run the operating system, receive requests from the network, prepare data, and hand the heavy maths to the AI chips, then collect the results. For example, NVIDIA’s DGX B200 server pairs 2 CPUs with 8 GPUs public.

So when you ask a chatbot something, CPUs do touch your request at the edges. But almost every one of the trillions of sums happens on the other kind of chip.

The whole picture

A few brilliant workers who hate waiting

A CPU core reads instructions one after another, keeps copies of what it needs close by, guesses ahead at every fork, works on whatever is ready while it waits, and does it all billions of times a second. A laptop has a handful to a dozen or so of these cores. That design is perfect for messy work and a poor fit for AI, which is one simple sum repeated beyond counting.

The obvious question is: what would a chip look like if you threw away the cleverness and filled the space with arithmetic instead? That chip exists, it was invented for video games, and it is part 5.