Inside LLM serving: one model, millions of chats

Typical time at each stage, as an example for a request with a few thousand prompt tokens:

The model has no memory between messages. Every turn, the app sends the instructions, tool descriptions, the whole conversation so far and your new message as one long text. The server rebuilds its working state from that text each time. Caches (chapter 13) make the rebuild cheap when the earlier part has not changed.

Streaming. Tokens are pushed to you the moment they are made, usually over a connection that stays open (server-sent events). So the first words appear right after prefill, not after the whole answer.

When it stops. The model writes a special end-of-turn token, or hits the output limit you set, or produces a stop phrase. The replica then frees that conversation’s seat, and the bill counts input and output tokens separately.

Scores, then chances. The last layer produces one raw score for every entry in the model’s word list, often 100,000 entries or more. Each score is divided by the temperature, then softmax turns the scores into chances. At temperature 0 the sampler simply takes the top word every time (called greedy decoding).

Cutting the tail. Before picking, servers often keep only the k best words (top-k), or the smallest group whose chances add up to some share p (top-p). This stops rare nonsense words from ever being chosen.

Why a longer answer costs more. Each new token needs one full pass through the model. A 1,000-token answer is 1,000 passes, one after another. They cannot run side by side, because token 501 depends on token 500. This single fact drives most of the engineering on the rest of the page.

Why answers vary. The pick uses a random number. Same prompt, different draw, different word, and every later word builds on it.

The six words and their scores here are made up to show the shape; they are not from a real model.

Building the list. Before training, a tokenizer is built by finding the chunks that appear most often in a huge sample of text. The common method is called byte-pair encoding: it starts from single bytes and keeps merging the most frequent pairs. Any text can still be written, piece by piece, so no word is ever “unknown”.

Why the count matters. Code, numbers and languages other than English usually need more tokens per word than plain English, so they cost more and fill the context faster.

Embeddings. The first step inside the model swaps each token’s id for a long list of numbers, looked up in a table with one row per token in the list. With 150,000 entries and 12,288 numbers per row, that table alone holds about 1.8 billion numbers.

Sizes in public models. public Llama 3 70B has 80 layers with 8,192 numbers per token; Llama 3.1 405B has 126 layers with 16,384; GPT-3 has 96 layers with 12,288; DeepSeek-V3 has 61 layers. not public Claude’s layer count and width.

Where the weights sit. Call the width of one token’s list d. Attention has four d-by-d tables (for the query, key, value and output), about 4d² numbers. A standard MLP widens to 4d and back, about 8d². That gives the one-third / two-thirds split. Modern designs shift the ratio a little, not the idea.

The cost rule that drives everything. Each weight is used for one multiply and one add per token. So one pass costs about 2 × (number of parameters) operations per token. A 200-billion-parameter model needs about 400 billion operations per token, and must read all 200 GB of its weights (at 1 byte each) to do it. Chapter 9 shows why that ratio decides the speed.

The running list is called the residual stream. Because each block only adds a correction, very deep stacks can still be trained, and early information is still available at the end.

Many heads. A layer runs dozens of these lookups side by side. Each one is called a head and learns its own view: one may track grammar, another which name a “she” refers to. Their results are joined and mixed by the output table.

Cost grows with context. Each new token compares its query with every earlier key, so the work per token grows in step with the conversation length. Across a whole sequence, the total grows with the square of the length. That is why very long contexts get expensive in both time and memory.

Only looking back. A token may only look at tokens before it (and itself). This rule, called the causal mask, is exactly what makes the saved keys and values reusable.

Word order. Attention by itself has no sense of order, so position is added in. The common method (rotary embeddings, or RoPE) turns each query and key by an angle that depends on its position.

200 billion parameters at 1 byte each (8-bit) is 200 GB. At 2 bytes (16-bit) it is 400 GB. At half a byte (4-bit) it is 100 GB. Lower precision trades a little accuracy for a lot of memory and speed (chapter 19).

Why only 60%. The rest of the memory is not spare. It holds every active conversation’s KV cache, temporary working numbers, buffers for talking to other chips, and the compiled programs. If the weights filled the chip there would be no room to serve anyone. The more room is left, the more people fit at once (chapter 11).

Why group sizes jump. Chips are sold in boxes of 8, linked at full speed inside the box, and a rack can link 72. Groups follow those physical shapes, so you round up to 8, 16, 32, 64 or 72 instead of using an exact count.

Mixture-of-experts models (chapter 16) store a large total but use only a slice per token. The total is what must fit in memory.

TB/s is terabytes per second; one PFLOPS is a thousand trillion (10¹⁵) operations per second. “Dense” means without a trick that skips zeros, which the headline spec-sheet numbers often include. NVIDIA’s DGX B200 page lists 180 GB per chip as usable, against 192 GB on the chip. Google TPUs and AWS Trainium chips have their own sizes in the same range. Anthropic has said it uses three chip platforms (Google TPUs, Amazon Trainium and NVIDIA GPUs), which is why this page says “chip” rather than “GPU”.

HBM is ordinary memory chips stacked in layers and mounted right beside the compute die, on the same package. That closeness is what gives it terabytes per second.

On-chip SRAM is far faster but tiny. An H100 has about 33 MB of registers, about 33 MB of shared memory and level-1 cache, and 50 MB of level-2 cache. Fast programs try to do as much work as possible on data already in SRAM, which is the idea behind FlashAttention (chapter 21). example The SRAM speed drawn is a rough total; vendors do not publish one number for it.

The arithmetic is done by tensor cores, units built to multiply small blocks of a table at once. A B200-class chip does about 4.5 thousand trillion 8-bit operations per second, while HBM delivers 8 trillion bytes per second: about 560 operations per byte.

Below HBM. The computer’s main memory, reached over a PCIe link, gives about 64 GB/s. A fast SSD gives around 7 to 14 GB/s. Servers can park rarely used KV caches there (chapter 13), but reading them back is far slower than HBM, so it is a fallback, not a plan.

Why output is priced higher. Reading a prompt token shares one trip to memory with thousands of other tokens. Writing a token shares its trip only with the other conversations running at that moment. Providers commonly price output several times higher than input for this reason.

Size of prefill. It costs about 2 × parameters operations per prompt token. For the 200B example, 3,000 tokens is 1.2 thousand trillion operations: about 70 ms of the whole 8-chip replica at the example’s 50% arithmetic efficiency. Real systems land at a few hundred milliseconds once attention, scheduling and communication are included.

Operations per byte. A chip can do a fixed number of operations per byte it reads, set by its hardware. For a B200-class chip that is about 4.5 thousand trillion operations per second divided by 8 trillion bytes per second: roughly 560 operations per byte. Writing one token for one person does about 2 operations per weight byte at 8 bits, 280 times below what the chip could handle. The arithmetic units are starved.

Arithmetic intensity is the name for operations done per byte moved. Decode for one person has an intensity near 2; prefill can reach thousands. A chip’s balance point is called its ridge. Below the ridge a job is limited by memory, above it by arithmetic. This picture is called the roofline model.

The fix is to raise the intensity. Serve B people from one read of the weights and the intensity becomes about 2 × B. With the example’s efficiencies (60% of peak memory speed, 50% of peak arithmetic) the balance point is about 230 people. Past it, the kitchen is the limit and extra people no longer come free.

Why not always run full? Each person then waits for a longer step, and all their KV caches must fit in memory. The next two chapters are about exactly that trade.

Each person gets one token per step, so their speed is 1 ÷ step, and the replica’s output is people ÷ step. For example, at 128 people and 8k tokens the step takes about 11.6 ms: about 86 tokens per second for each person and about 11,000 for the replica.

What the example leaves out. Real systems add scheduler delays, uneven expert loads, time spent picking tokens, and prompt-reading work that interrupts writing. They also use tricks such as speculative decoding (chapter 20) that change the curve.

The example model has 80 layers, 8 KV heads and 128 numbers per head at 1 byte: 2 × 80 × 8 × 128 × 1 = 163,840 bytes, about 160 KB per token. The 2 is for keys and values. Multiply by the conversation length to get the pile’s size.

Where the saving comes from. Without a cache, writing token n means rebuilding keys and values for tokens 1 to n, so the total work over an answer grows with the square of its length. With the cache, each step builds exactly one new pair. The price is memory that grows steadily with length for as long as the conversation stays active.

Reading it back is the real cost. Every writing step reads every person’s whole cache from memory, on top of reading the weights once. At long lengths the cache reads take most of the step.

Memory is the cap. People per replica ≈ (usable memory − weights) ÷ (cache size of one conversation). At 8k tokens the example fits about 900, at 128k about 56, and at 1M tokens about 7. The arithmetic might support more, but the memory does not.

• MHA. Every head stores its own keys and values. Biggest cache: 2 × 64 × 128 = 16,384 numbers.
• MQA. One key-value pair shared by all heads. Smallest cache (2 × 1 × 128 = 256), with some loss of quality. Proposed by Noam Shazeer in 2019.
• GQA. Heads are split into groups, one pair per group; with 8 groups that is 2 × 8 × 128 = 2,048. A middle path that keeps most of the quality. public Used in Llama 2 70B and Llama 3, among others.
• MLA. Used in DeepSeek-V2 and V3. public DeepSeek-V3 keeps 512 compressed numbers plus 64 for position: 576 per token per layer.

Other ways to shrink it. Some layers only look at the last few thousand tokens (a sliding window), so their cache stops growing. Keys and values can be stored in 8-bit or 4-bit instead of 16-bit. Tokens that attention rarely uses can be dropped or compressed. Some layer types keep a fixed-size summary instead of a growing list, and are often mixed with a few ordinary attention layers.

not public Which of these Claude models use. The point is the pressure: every byte shaved from the cache is room for more people per replica.

Prompt caching. Blocks are labelled by a fingerprint (a hash) of their contents plus everything before them. A new request that matches reuses the blocks and skips reading that part of the prompt. This is the mechanism behind the prompt caching that API providers sell: cached input tokens are cheaper because they cost memory, not arithmetic.

Why caches expire quickly. A cached prefix takes scarce HBM that could hold live conversations. Servers throw out the blocks used least recently, so entries typically live minutes, not hours, unless moved to a slower tier.

Tiers. Cold blocks can move from HBM to the computer’s main memory, to SSD, or to a shared pool over the network; the public Mooncake system describes this. Reading them back is sometimes slower than recomputing, so schedulers choose.

When memory fills. The scheduler may pause a conversation: move its blocks out, or drop them and recompute later. Good scheduling keeps this rare.

not public Anthropic’s internal design. Prompt caching is a documented product feature; the machinery here is the public research it most resembles.

An all-reduce means every chip sends its partial result and every chip gets back the sum. An all-to-all means every chip sends something to every other chip.

How they combine. A common shape is tensor splitting inside one 8-chip server or one 72-chip rack, experts spread across a larger group, and many identical replicas behind a router. DeepSeek’s public description of its serving system spreads experts across dozens of chips, so each chip holds only a few and reads fewer weights per step.

Speed or capacity? Tensor splitting divides the bytes each chip reads, so each step gets faster. Pipeline splitting does not: one token still walks through every stage in turn. It lets a model fit across more chips with cheaper links, at the cost of “bubbles”, moments when a stage has nothing to do.

Splitting the conversation. For very long prompts, the sequence itself can be divided among chips (sequence or context parallelism), so no single chip holds the whole KV cache.

not public The mix Anthropic uses.

The numbers. In the 128-person example, each all-reduce moves about 3 MB (128 people × 12,288 numbers × 2 bytes). There are two per layer and 80 layers, so 160 per step, around 500 MB. Over 900 GB/s NVLink that takes about half a millisecond. Over a 50 GB/s network link it takes about ten milliseconds, nearly a whole step. Each exchange also has a fixed delay of several microseconds, which is why the example charges 2 ms of overhead per step even on fast links.

Link domains. public An 8-GPU server links its chips with NVLink. NVIDIA’s GB200 NVL72 rack links 72 Blackwell GPUs in one NVLink domain, about 130 TB/s combined, which lets a single replica span a whole rack. Google TPU pods link thousands of chips with their own fast links.

Between servers. Servers connect through InfiniBand or Ethernet with remote memory access, so one chip can write straight into another server’s memory without involving its main processor. Switches are arranged so any pair of racks has plenty of paths.

What never travels: the weights. They are loaded once when a replica starts and stay put. What moves while serving is small by comparison: numbers between chips, requests and tokens to and from the router, and sometimes KV cache blocks between prefill and decode pools (chapter 18).

The internet leg. Your connection carries only text. A 1,000-token answer is a few kilobytes, so streaming it adds almost nothing next to the time spent generating it.

Total versus active. public DeepSeek-V3 has 671 billion parameters in total and 37 billion active per token, with 256 routed experts per layer and 8 chosen for each token. Mixtral 8x7B has 8 experts per layer and uses 2.

The router. A small table scores every expert for each token. The top few are picked and their outputs are blended using those scores. Tokens in the same sentence can go to different experts.

Why batching changes the picture. With one token, only 8 of 256 experts are read, about 3% of the expert weights. With 64 tokens, the chance that a given expert is untouched is (1 − 8/256)⁶⁴, about 13%, so about 87% are read anyway. With 256 tokens nearly all of them are. At the batch sizes that make serving efficient, the memory reading costs about as much as a dense model of the same total size, while the arithmetic stays at the small active size.

Why spread experts over many chips. If 256 experts live on 32 chips, each chip holds 8 and reads only those 8 per step. Each chip serves a large pool of tokens sent from everywhere, so every expert it holds sees a healthy batch. That is expert parallelism (chapter 14); its cost is an all-to-all exchange of tokens at every MoE layer, where every chip sends to every other chip.

Weak spots. Some experts get more traffic than others, and the slowest chip sets the step time. Training adds balancing rules to keep load even; serving adds extra copies of popular experts. not public Whether any Claude model is dense or mixture-of-experts.

The lit squares come from a fixed pseudo-random routing, so the count can differ a little from the formula.

Two limits on joining. Free KV memory (chapter 11) and a speed promise. If admitting more people would stretch each step past the speed promised to users, the scheduler keeps them in the queue. This is why overload first shows up as waiting rather than errors, until a queue limit is reached.

Mixed steps. One step can include writing work for running conversations and a slice of prompt reading for a newcomer. Chapter 18 shows why that mixing needs care.

What this means for you. Your answer’s speed depends on who happens to share your step. That is one reason the same prompt can stream at different speeds at different times of day.

Chunked prefill was published as Sarathi-Serve (2024) and is available in open serving software.

Separate pools (called disaggregation) appear in public work such as Splitwise (Microsoft), DistServe and Mooncake (Moonshot AI). Each pool can use its own chip count, batch size, even chip type. Mixing and chunking are simplest; separate pools win at large scale.

The balancing act. Too few prompt-reading chips and the first token is late; too few writing chips and answers stream slowly. The fleet adjusts the ratio as traffic changes, for example when coding tools send long prompts and get short replies, or when people ask for long essays from short prompts.

not public Which of these arrangements Anthropic runs.

Why speed follows bytes. A memory-limited step takes weights ÷ memory speed: half the bytes, half the time. Newer chips also do low-precision arithmetic faster. public A B200-class chip is rated at about 2.25 thousand trillion operations per second at 16-bit, 4.5 at 8-bit and 9 at 4-bit, without the zero-skipping trick.

Scaling in blocks. Weights are grouped, for example 16 or 32 at a time, and each group shares a scale factor stored in a few extra bits. Formats such as MXFP4 and NVFP4 are built this way.

What gets quantized. Weights always. Often the KV cache too (chapter 12). Sometimes the temporary working numbers as well, which lets the arithmetic itself run at low precision.

Where quality can slip. A few weights and working numbers are unusually large and matter a lot; rounding them hurts. Methods handle them by rescaling, rotating, or keeping them at higher precision. The checks are benchmarks and, more importantly, comparing outputs before and after.

not public The precision Claude models are served at.

Here k is the draft length, a is the chance each draft token is accepted, and draft cost is a draft step as a share of a big step. The first formula counts the accepted tokens plus the one the big model always supplies where it first disagrees. It treats each guess as accepted independently, which is a simplification.

The output does not change. The standard method (Leviathan and colleagues, 2022; Chen and colleagues, 2023) accepts or rejects guesses with a rule that provably leaves the big model’s output exactly as likely as without the trick. It is a speed trick, not a quality trade.

When it helps less. With many people on the step, the arithmetic units are already busy, so extra checked tokens are no longer free. It shines when few people share a step, as in low-latency products. Easy text (boilerplate code, quotes, repeated phrases) is accepted far more often than hard reasoning.

Variants. Draft with a tiny model; draft with extra prediction heads on the big model itself (Medusa, EAGLE, multi-token prediction); or draft by copying phrases that already appeared in the prompt.

Kernel fusion. Many tiny steps around the big multiplications (normalising, adding to the running list, applying the activation, picking the token) each read and write the whole list of numbers. Fusing them into one kernel keeps the data on the chip between steps.

Launch cost. Starting each kernel costs microseconds, and one writing step has thousands of small kernels. Recording them once and replaying them (CUDA graphs on NVIDIA, compiled graphs on TPUs) removes most of that cost.

Overlap. While one group of chips computes, the previous result is already being sent over the wire. Good schedules hide most of the communication from chapter 15 behind arithmetic.

Limits protect the fleet. Per-account limits on requests and tokens per minute stop one customer from soaking up shared capacity. They also help planning: the provider knows the most traffic any tier can send.

Priority. Live chat is sensitive to delay, so it goes first. Bulk jobs that can wait run in the gaps; Anthropic offers its batch API at a discount for exactly this. That is how a fleet keeps chips busy without hurting live users.

Cache-aware routing. Sending a conversation back to the replica that still holds its KV blocks saves reading the earlier part again. Routers trade off load against cache hits, like “sticky sessions” on a website. A real router also weighs geography, model version and failures.

Shedding. Under heavy load, a system can protect core traffic by refusing lower-priority requests first, shortening allowed outputs, or both.

Updates without downtime. A new model version is loaded onto spare replicas, tested on a trickle of traffic (a “canary”), then given a growing share while old replicas finish their conversations and drain. Loading weights takes minutes, so rollouts are planned, not instant.

Many models at once. A fleet serves several model sizes and versions at the same time, each with its own replicas, memory layout and routing rules. Smaller tiers fit in fewer chips and serve more people per chip, which is part of why they cost less.

Many kinds of chips. public Anthropic has said it uses Google TPUs, Amazon Trainium and NVIDIA GPUs. The same model has to be compiled and tuned separately for each, and traffic is spread across them.

Demand moves. Traffic follows working hours around the world, peaks on weekdays, and can surge when a new feature launches. Capacity is planned months ahead, because racks cannot be rented by the minute at this scale.

Prompt-reading time comes from the tokens that are not cached, at about 2 operations per parameter each, plus attention work. Step time is the chapter 10 model at 128 people per replica, lowered when long conversations do not fit in memory. 128 is an assumed operating point, not a measured one.

What moves the answer most. The cached share (cached tokens skip prompt reading entirely), the conversation length (it sets both KV memory and the cost of each step), and the length of replies. AI agents and long reasoning make replies longer and keep conversations open longer.

Why list prices are far above the chip cost. The calculator counts only chip rental. Real prices also pay for idle capacity kept for peaks and failures, networking and buildings, much larger models, lower real efficiency than an ideal model, engineering, and training the model in the first place. Do not read the last tile as a price.

not public None of the defaults are Anthropic’s numbers.