# Transformer Architecture

How a Transformer translates an English sentence into German, from turning words into numbers to attention, layered encoding and writing the output\.

By Christian Langreiter

Topic: Computing, AI & Electronics

[Open the interactive Jig](https://ask.jig.com/j/ttF8fOPLhlK6)

## Transcript

### 1. A machine that reads

This glowing model is a Transformer, a design for computer programs that work with language\. It was introduced in 2017 to translate sentences\. Here it turns an English sentence into German\. Drag the scene to look around\.

The left tower is the encoder, which reads\. The right tower is the decoder, which writes\.

### 2. Words become numbers

A computer cannot read letters, so each word is swapped for a long list of numbers called an embedding\. Words with similar meanings get similar lists\. The little bars on each tile stand for that list\.

In the original model every word becomes 512 numbers\. We show 8 of them\.

### 3. Where is each word?

The whole sentence goes in at once, not one word after another\. So the model adds a pattern of waves to each list of numbers\. The waves give every position its own mix of values, which tells the model the word order\.

Each wave has a different length, like the hands of a clock turning at different speeds\.

### 4. Paying attention

Attention is the big idea\. Every word scores how useful every other word is to it, and the scores are shared out so they add up to 100%\. Thicker arcs mean more attention\. Pick a word to see where it looks\.

To translate "it" into German, the model must work out that "it" means the cat\. These arcs show the kind of pattern a trained model can learn\.

### 5. Eight heads at once

One round of attention is not enough, so the model runs eight side by side\. Each one is called a head\. One head might link "it" to "cat" while another looks at the next word or the word before\.

Each head works with a short list of 64 numbers, and the 8 results are joined back into 512\.

### 6. Think, then tidy

After attention, every word passes through its own small network, called feed\-forward, that refines what it has learned\. Shortcuts around each part add the word's earlier numbers back in\. A tidy step then keeps the values in a steady range\.

Inside the feed\-forward part, 512 numbers widen to 2,048 and then shrink back to 512\.

### 7. Six layers deep

The encoder repeats attention and feed\-forward in six identical layers\. Each layer sharpens the meaning of every word a little more\. At the top, each word carries what it means in this whole sentence\.

The whole base model has about 65 million adjustable numbers, learned from millions of example sentences\.

### 8. Writing the answer

The decoder writes the German sentence one word at a time\. Each new word may look back at the words already written, never ahead\. It also looks across at the encoder's finished meaning of the English sentence\.

Because "Katze" is a feminine word in German, "it" becomes "sie"\. Attention to "cat" makes that possible\.

### 9. Why it changed everything

Older designs read words one at a time\. A Transformer handles all of them together, so it can learn fast on many chips at once\. Many of today's AI chat assistants are built from the decoder half of this design\.

The base model learned to translate in about 12 hours on 8 graphics chips\.