# CUDA & AI Training at Scale Fly from a data hall down to one GPU die to see why AI training needs thousands of networked GPUs and why CUDA matters, then tune GPUs, network, scale, software and power price to see time to train and cost per token By Ham Sandwich Topic: Computing, AI & Electronics [Open the interactive Jig](https://ask.jig.com/j/bJPL0KLRo3Fd) ## Guide ### 1. The machine that learns This is an AI training hall: 72 racks of computers, wired together and drawing as much power as a small town\. Big AI models are not programmed line by line\. They learn by reading trillions of words and nudging billions of numbers, over and over, until their guesses get good\. That takes so much arithmetic that one computer would need thousands of years\. So we build halls like this and make the computers work as one\. Drag anywhere to turn the hall\. Use the arrows to fly closer\. ### 2. One GPU, up close Inside each server sit eight GPUs, graphics processing units\. This one is an NVIDIA H100\. Its copper and aluminium cooler is lifted off so you can see the chip package: one large silicon die in the middle, ringed by six stacks of fast memory called HBM\. Around it, the black blocks are power stages that turn 54 volts into the less than one volt the chip runs on, delivering up to 700 watts\. 80 billion transistors on a die about the size of a postage stamp\. ### 3. Many hands: parallel computing An ordinary processor is like a few brilliant workers doing one job after another\. A GPU is like a stadium full of workers who each do a simple sum at the same moment\. The H100 die has 132 blocks called streaming multiprocessors, the glowing tiles, with 16,896 small cores between them\. Training AI is mostly multiplying huge tables of numbers, and every piece can be done at once\. Switch between one worker at a time and all of them at once\. ### 4. CUDA: speaking the GPU's language Thousands of workers are useless unless someone can tell them what to do\. CUDA, released by NVIDIA in 2007, is that language and the tools around it\. A programmer writes one small function, a kernel, and CUDA launches it on thousands of threads in groups of 32\. Above the hardware sit four layers: the driver, the CUDA runtime and compiler, ready\-made math libraries, and the AI frameworks researchers actually use\. Blue: driver\. Teal: runtime and compiler\. Green: libraries\. Amber: frameworks\. ### 5. Why one GPU is not enough A model like Llama 3 405B has 405 billion numbers to learn\. Just storing them takes about 810 gigabytes, ten times what one GPU holds\. So the model is cut into pieces spread across many GPUs\. After every small batch of training, all GPUs must swap what they learned, called gradients, before anyone can continue\. Here, NVLink and four NVSwitch chips let all eight GPUs in this server share data at 900 gigabytes a second each\. Click any GPU to light it up, or sync them all\. ### 6. Racks, power and the network Each server draws about 10 kilowatts, so a rack of four needs more power than a dozen homes\. Amber light shows power flowing from the copper busbar overhead\. Cyan light is the network: eight 400 gigabit links leave every server through switches and yellow fibre trays to reach every other rack\. Training is a rhythm: everyone computes, then everyone talks\. A slow network leaves thousands of GPUs idle, waiting\. Meta trained Llama 3 on 16,384 H100s at once, for about two months\. ### 7. Why CUDA is hard to copy Other chips can do this math too\. AMD offers ROCm, and open standards like OpenCL and oneAPI aim to run anywhere\. What sets CUDA apart is nearly two decades of building: tuned libraries such as cuBLAS, cuDNN and NCCL, the code that makes thousands of GPUs exchange gradients in lockstep, debugging tools, and millions of developers\. AI frameworks were built on it first, so new models usually run fastest on CUDA on day one\. Watch every rack pulse together: compute, then sync\. ### 8. Playground: choose your GPUs Now you run the project\. You are training a 405 billion parameter model on 15\.6 trillion tokens of text, like Llama 3\. Pick a GPU generation\. Newer chips do more math per second and per watt, so the same work finishes sooner and costs less, even though each chip costs more\. The readout at the top updates as you change things\. Estimates from public figures; real projects vary\. ### 9. Playground: wire the network Choose how the racks are connected\. With slow 100 gigabit Ethernet, GPUs spend much of each step waiting for gradients\. 400 gigabit links with RDMA, either RoCE Ethernet or InfiniBand, let data move straight from GPU memory to GPU memory\. Meta used RoCE for its largest runs; InfiniBand is a little faster but costs more\. Watch the cyan traffic and the time spent waiting\. ### 10. Playground: scale it out Slide from about a thousand GPUs to 65,536\. Twice the GPUs nearly halves the time, but not quite: every doubling adds more talking for each step, and with more machines something breaks more often and the job must restart from a checkpoint\. Each glowing rack here stands for about 900 GPUs\. Power demand climbs past 100 megawatts at the top end\. ### 11. Playground: tune the software Same hardware, different code\. Tuned CUDA libraries keep the cores fed and hide the network behind the math\. Naive kernels, code that works but is not optimised, leave most of the chip idle\. This gap is a big part of why CUDA's long\-built software matters: it can be worth as much as a whole new generation of chips\. ### 12. Playground: pay for power Finally, the electricity bill\. Industrial power costs from about 4 to 40 cents per kilowatt hour depending on where you build\. For most runs, electricity is only 10 to 15 percent of the cost; the GPUs themselves are the big expense\. That is why the fastest, best\-used GPUs win: every hour they save is an hour of very expensive hardware freed up\. You can go back to any step and change it again\.