# Inside LLM Inference

By Umesh Yadav

Topic: Computing, AI & Electronics

[Open the interactive Jig](https://ask.jig.com/j/g1wIaQll93MA)

## Guide

### 1. From keystroke to tokens

Here is a real\-feeling GPU server, the kind that runs a chatbot\. Type a prompt and watch it break into tokens: small pieces of text, often a whole word, sometimes part of one\. The model never sees letters, only these pieces\.

In English, one token is about three quarters of a word\.

### 2. Tokens become numbers

Each token is swapped for an embedding: a long list of numbers that captures its meaning\. The lists fly into a stack of transformer layers, the model's processing floors\. Every layer refines the numbers and passes them up\.

A 70\-billion\-parameter model uses 8,192 numbers per token and 80 layers\. We show 12\.

### 3. Inside one layer

Each token makes three new lists: a Query \(what am I looking for?\), a Key \(what do I offer?\) and a Value \(what do I share?\)\. Attention heads match Queries to Keys and glow brighter where tokens matter to each other\. Then the MLP, a small number mixer, stretches every list wide and squeezes it back\.

Tokens only look back at earlier tokens, never ahead\.

### 4. Open the GPU

Now lift one GPU out of the server\. Beside the chip sit HBM stacks, towers of fast memory that hold the model's weights\. On the chip, a band of L2 cache feeds 132 SMs, small processors, each with four tensor cores built for multiplying grids of numbers\. Watch weights stream from HBM into the SMs' tiny on\-chip memory\.

H100: 80 GB of HBM3 moving about 3\.35 terabytes per second\.

### 5. Waiting for memory

Prefill reads your whole prompt at once, so every weight fetched from memory is used for many tokens: the math units stay busy\. Decode writes the answer one token at a time, so each weight is fetched for very little math and the chip mostly waits\. The roofline chart shows the speed limit: the slanted roof is memory, the flat roof is math\.

Decoding one chat at a time uses well under 1% of the chip's math power\.

### 6. The KV cache

To avoid redoing work, the server keeps each token's Keys and Values in a KV cache that grows with every new token\. Old servers reserved one long block per chat, leaving gaps when chats ended\. PagedAttention stores the cache in small pages of 16 tokens, so the gaps disappear\.

Older systems wasted 60 to 80% of cache memory; paging cuts waste to about 4%\.

### 7. Batching chats together

Reading the weights once can serve many chats at the same time, so servers batch requests\. Static batching waits for the longest answer before anyone new can start\. Continuous batching slips a waiting chat into each free seat right away\.

Each lane is one seat in the batch; the stack on the left is the waiting queue\.

### 8. Smaller numbers

Quantization stores each weight with fewer bits: 16\-bit FP16, 8\-bit FP8 or 4\-bit INT4\. The model shrinks to a half or a quarter, so it reads faster and fits on fewer GPUs\. Quality wobbles only a little, shown by the perplexity meter, where lower means the model is less surprised by real text\.

Perplexity for a 65B model: 3\.53 at FP16, 3\.85 at INT4 \(FP8 shown is an estimate\)\.

### 9. A quick draft, one check

Speculative decoding lets a small draft model guess the next few tokens quickly\. The big model checks all the guesses in a single pass, keeps the ones it agrees with and fixes the first mistake\. The answer is exactly what the big model would write, just sooner\.

Typical speed\-ups are two to three times\.

### 10. Eight GPUs, one model

A 70B model at FP16 needs about 140 GB, more than one GPU holds, so it is split across all eight\. Tensor parallel cuts every layer into eight slices and the GPUs swap results after each layer\. Pipeline parallel gives each GPU its own 10 layers and passes the work along\. Watch the NVLink traffic through the four switch chips\.

NVLink moves up to 900 GB/s per GPU, counting both directions\.

### 11. Turn the dials

Now run the whole server\. Drag batch size, context length and model size, and watch time to first token, tokens per second, cost and spare memory change\. Answers stream back to you one token at a time\.

Numbers come from a simple estimate for 8 H100s at an example price of $3 per GPU\-hour\.