# How Transformers Work

A word machine shows how a transformer language model turns a sentence into tokens, numbers, attention and a guess for the next word

By quiet\-dolphin\-7897cb5f

Topic: Computing, AI & Electronics

[Open the interactive Jig](https://ask.jig.com/j/5VlMWJY5QyYu)

## Transcript

### 1. A machine that guesses the next word

Chatbots are built on a design called the transformer\. These big language models, or LLMs, do one thing over and over: guess what comes next\. Here our word machine reads “The cat sat on the” and gets ready to guess\.

Drag the room to look around\. Pinch or scroll to move closer\.

### 2. Cutting text into tokens

First the text is cut into small pieces called tokens\. A token can be a whole word or just part of one\. Each token is looked up in a fixed list, the vocabulary, and gets its own number\.

GPT\-3’s vocabulary held about 50,000 different tokens\.

### 3. Words become lists of numbers

Each token is swapped for a long list of numbers, a bit like an address for its meaning\. Words with similar meanings get similar lists\. Warm cells here stand for positive numbers and cool cells for negative ones\.

The original 2017 transformer used 512 numbers per token\. We show just 8\.

### 4. Remembering word order

On their own, these lists would not say which word came first\. So the machine adds a position signal, made from smooth waves, to every list\. Now “cat sat” and “sat cat” look different\.

### 5. Paying attention

Now each token looks back at the words before it and decides which ones matter\. This step is called attention\. Brighter links mean stronger attention, and meaning flows along them into the token\.

Tap any word tile to see its links\. A token can only look back, never ahead\.

### 6. Many heads, many questions

Attention runs as several “heads” side by side, each with its own colour here\. Each head learns to look for a different kind of link, such as which word came just before, or who is doing what\.

The original transformer had 8 heads in each layer\. GPT\-3 had 96\.

### 7. Up through the layers

After attention, a small network called feed\-forward polishes each token on its own\. Attention plus feed\-forward make one layer, and the layers are stacked like floors\. Each floor builds a richer picture of the sentence\.

The original had 6 layers in each half\. GPT\-3 had 96\.

### 8. Picking the next word

At the top, the machine scores every token in its vocabulary and turns the scores into chances that add up to 100%\. It picks one, adds it to the sentence and runs again, one token at a time\.

The chances shown are a made\-up example\. It does not always pick the top word, so its writing stays varied\.

### 9. Learned from practice

Nobody writes these rules by hand\. In training, the model guesses the next token across huge amounts of text and nudges its numbers, called parameters, after every miss\. It predicts likely words; it does not look up stored sentences\.

GPT\-3 had about 175 billion parameters, roughly 2,700 times more than the original transformer\.