# Attention Is All You Need How the 2017 Transformer reads a sentence in parallel, scores which words attend to which, and translates English into German\. By merry\-dolphin\-b42a718a Topic: Computing, AI & Electronics [Open the interactive Jig](https://ask.jig.com/j/NTUTN7g2xqPp) ## Transcript ### 1. Attention Is All You Need In June 2017, eight researchers working at Google published a short paper with a bold title\. It introduced the Transformer, a network that reads a whole sentence at once and lets every word look at every other word\. The chatbots, translators and image models of today are built on it\. The title nods to the Beatles song All You Need Is Love\. Drag empty space to turn the scene\. ### 2. Where neural nets came from An artificial neuron is simple: it adds up weighted inputs and passes a signal on\. Stacked in layers and trained with backpropagation, which nudges every weight to shrink the error, such networks learned to recognise images\. Language was harder, because words arrive in order and sentences vary in length\. The perceptron of 1958 had a single layer; deep networks only took off in 2012\. ### 3. Reading one word at a time Before 2017 the best translators were recurrent networks\. They read word 1, then word 2, then word 3, squeezing everything so far into one running memory\. Each step waits for the one before, so the work cannot be spread across thousands of chip cores, and early words fade by the end\. Compare the two ways of reading\. The paper's Table 1: a recurrent layer needs n steps in a row for n words; self\-attention needs one\. ### 4. Words become numbers A Transformer first turns each word into a list of 512 numbers, its embedding, learned so that words used alike get similar lists\. Attention on its own ignores word order, so the model adds a position signal built from sine and cosine waves of many wavelengths\. Switch the waves on to see each column shift\. Learned position embeddings worked almost as well; the waves might cope with longer sentences\. ### 5. Query, key and value Each word makes three vectors: a query \(what am I looking for?\), a key \(what do I offer?\) and a value \(what I pass on\)\. One word's query is scored against every key, softmax turns the scores into weights that sum to 1, and the word takes that weighted mix of values\. Tap any word to see where it looks\. Attention\(Q, K, V\) = softmax\(QKᵀ/√dₖ\)V\. Dividing by √dₖ stops the scores growing too large\. Weights here are illustrative\. ### 6. Many heads at once One attention pattern is not enough, so the Transformer runs 8 heads side by side, each in its own 64\-number slice, then joins their results\. In the paper's own pictures, heads appear to learn different jobs: one links its to Law and application, others seem to follow the structure of the sentence\. With a single head, translation quality dropped by 0\.9 BLEU \(Table 3\)\. ### 7. The Transformer stack Stack it up\. Six encoder layers each pair self\-attention with a small feed\-forward network\. Six decoder layers add masked attention and attention over the encoder's output\. Around every sublayer a shortcut adds the input back and normalises the sum, which keeps a deep stack trainable\. Base model: 512\-number vectors, 8 heads, about 65 million parameters\. ### 8. Translate, one word at a time The encoder reads the whole English sentence in parallel\. The decoder then writes German one word at a time, attending to the encoder's output and to the words it has already written\. A mask hides later positions, so during training it cannot peek at the answer\. Trained on about 4\.5 million English\-German sentence pairs\. ### 9. Better, faster, everywhere Trained for 3\.5 days on 8 GPUs, the big Transformer scored 28\.4 BLEU on English to German, over 2 points above the best earlier systems, ensembles included, for a fraction of their training cost\. Its encoder later grew into BERT and its decoder into GPT; ChatGPT followed in 2022\. BLEU measures overlap with human reference translations: higher is better\.