# How Foundational Models Process Data

Follow text and video through a transformer foundation model: tokens, patches, embeddings, attention, MLP layers and the next\-token readout

By quiet\-wombat\-f8bbf225

Topic: Computing, AI & Electronics

[Open the interactive Jig](https://ask.jig.com/j/OqTmygiVkrq1)

## Transcript

### 1. Inside a foundation model

A foundation model is one large network trained on huge amounts of text, images and video\. Whatever it reads becomes rows of numbers that rise through a tower of identical layers\. At the top, those numbers become a guess at what comes next\. Follow one short video clip and one sentence all the way through\.

Drag anywhere off the model to turn the view\.

### 2. Text becomes tokens

A tokenizer cuts text into common pieces called tokens: whole words, parts of words or punctuation\. Here "bounced" splits into "bounc" and "ed"\. Each piece is swapped for its number in a fixed vocabulary, so the network never sees letters, only these IDs\.

Vocabularies hold roughly 32,000 to 200,000 pieces\. The IDs shown are examples\.

### 3. Video becomes patches

Video is not watched like a film\. Each frame is cut into a grid of small square patches, and patches at the same place in neighbouring frames are bundled into one block called a tubelet\. Every tubelet becomes one token, so four frames here turn into eight video tokens\.

Real models use patches of about 16 by 16 pixels, so even a short clip makes thousands of tokens\.

### 4. Tokens become vectors

Each token ID picks out one row of a learned table: a list of numbers called its embedding\. Video tubelets are flattened and projected into the same kind of list\. A small position signal is added so the model knows the order\. Now text and video sit side by side as one sheet of numbers, one row per token\.

Each row here shows 8 numbers\. Large models use thousands per token\.

### 5. Attention: tokens share information

In each layer, attention lets every token gather information from the tokens before it\. Each row scores how relevant the earlier rows are, then adds in a weighted mix of what they carry\. Thicker lines mean more weight\. Click any row to see where it looks\.

A token can only look back, never ahead, because the model predicts what comes next\.

### 6. The MLP: each token thinks alone

Next comes a feed\-forward network, or MLP\. Each row is processed on its own: its numbers are spread into a layer about four times wider, only some of those hidden units switch on, and the result is squeezed back down and added to the row\. Much of what the model has learned is stored in these weights\.

### 7. Many layers deep

One attention step plus one MLP make a layer, and the same pattern repeats again and again\. The rows are never taken out and rebuilt: each layer adds a little more to the same running sheet\. Early layers tend to track nearby words and shapes; later ones carry more abstract meaning\.

Four layers are shown\. GPT\-3, for example, stacks 96\.

### 8. Reading out the next token

After the last layer, only the final row is read out\. It is compared with every entry in the vocabulary to give a score for each, and the scores become probabilities\. The model then picks one token, adds it to the end of the row, and runs everything again to choose the next\.

The candidates and percentages are examples\.

### 9. Reassembled, not retrieved

Nothing was looked up in a library of stored sentences or clips\. The answer was assembled on the spot as information flowed between rows and through billions of learned weights\. Many models that make video use a similar stack of layers, then a decoder turns the predicted patches back into pixels\.

GPT\-3 has 175 billion learned weights\.