How AI works

You type a question: “Why is the sky blue?” To a computer it is a string of characters. A language model works with a fixed set of pieces instead, so the first job is to find them.

A tokenizer splits the text into tokens: common words whole, rarer words in parts, punctuation on its own. The space before a word usually travels with it, which is why “ sky” and “sky” can be different tokens.

Every token has a number: its place in the model’s vocabulary, a list fixed before training begins. Vocabularies hold from tens of thousands to a few hundred thousand tokens. The numbers here are illustrative; each tokenizer has its own.

Your question is now a row of whole numbers. From here on, everything the model does to it is arithmetic.

A token’s number is only a label. The model swaps each one for a vector: a list of numbers it learned during training. Our six tokens become six columns of values.

Read as coordinates, each vector is a point. Training arranges the points so that words used in similar ways end up near each other. The picture squeezes thousands of dimensions into three.

“sky” lands among cloud, air and rain; “blue” among the other colours. Nobody wrote those rules down: they come from how words are used across the training text.

In a well-known early result, the step from “man” to “woman” matched the step from “king” to “queen”. Directions in the space can carry ideas like these. The layout here is illustrative.

A neuron takes a few numbers in and sends one number out. This one has four inputs.

Each input is multiplied by a weight. A large weight makes its input count for a lot; a negative weight pushes the other way. The weights are what training adjusts.

The neuron adds the products and a bias, then passes the sum through an activation function: a bend that lets a network describe more than straight lines. GELU, drawn here, is common in language models.

A model is built from enormous numbers of these simple parts. Its parameters, the weights and biases, are counted in billions.

The token vectors enter the first layer.

Every neuron in a layer reads the outputs of the layer before, weighs them, adds them and bends the sum. The signal sweeps forward, one layer at a time.

At the far end, the last layer produces the numbers the model will choose from.

Stacking many layers is what makes a network deep. Language models are much deeper than this sketch, and their layers are of a particular kind: transformer blocks.

On its own, “blue” could be a colour or a mood. To know which, the model lets each position gather information from the positions before it.

“blue” scores every earlier word by how relevant it is, then takes a weighted mix of what those words carry. Here it leans most on “sky”.

An attention layer runs several of these side by side, each with its own weights. One head may follow what is being described, another the word just before, another the question being asked.

As a grid, each row is one position’s attention, adding up to one. The upper half is empty: in a language model a word can look back, never ahead.

A transformer block has two parts. Attention lets positions exchange information; a feed-forward network then works on each position by itself.

Blocks are stacked, each refining what the block below produced.

Through the middle runs a stream of numbers for every position. Each block reads from it and adds its own contribution back, so information climbs without being lost.

After the last block, the stream at the final position becomes a score for every token in the vocabulary.

The tower gives a score to every token in the vocabulary: tens of thousands of numbers.

Softmax turns the scores into probabilities that add up to one. Temperature sets how bold the choice is: low temperature sharpens the top candidates, high temperature evens them out. Try the slider.

A token is drawn at random, weighted by those probabilities. Usually a likely token wins; sometimes a less likely one does, which is why the same question can get different answers.

The chosen token joins the text, and the whole model runs again to choose the next. The answer grows a token at a time until the model chooses to stop.

Training begins with random weights, so the model’s predictions are poor. Loss measures how poor; here, it is the height of the landscape.

For every weight, calculus gives the gradient: the direction in which the loss rises fastest. Training takes a small step the opposite way.

Repeating the step walks the weights downhill. Not every walk ends in the deepest valley: this one does, while a second walker settles in a shallow dip.

A real model walks through billions of dimensions at once, not two, one batch of training text after another.

The model’s prediction is compared with the right answer: the token that really came next in the training text.

Starting at the output, the error is passed backwards through the network, layer by layer, with the chain rule of calculus.

Each weight receives its share of the blame and moves a little in the direction that would have made the error smaller.

Training alternates the two passes, again and again, across all its text.

A model trained only to predict the next token of web text is a base model. Asked a question, it may simply carry on the page: more questions, a list, a heading.

People write example answers to many prompts, and the model is trained on them the same way as before, one token at a time. This is supervised fine-tuning.

The model writes several answers and people rank them. A second model, the reward model, learns to predict which answers people prefer.

Reinforcement learning then adjusts the model to earn higher scores from the reward model. Newer methods, such as direct preference optimisation, learn from the rankings without a separate reward model.

Most of a layer’s work is one calculation: a matrix of inputs multiplied by a matrix of weights.

Each output cell pairs a row of inputs with a column of weights, multiplies them pair by pair and adds the results. The first cell here takes six multiplications.

No cell depends on another, so they can all be worked out at the same moment. Hardware for AI is built around that independence.

A real layer multiplies matrices with thousands of rows and columns. Writing one token takes roughly two operations for every parameter in the model.

A CPU has a handful of powerful cores, built to race through complicated tasks one after another.

Given thousands of independent sums, a CPU works through them a few at a time.

A GPU has thousands of simpler cores that run the same instruction on different numbers at once. Designed to colour the pixels of games, the idea suits neural networks just as well.

Modern GPUs add tensor cores: units that multiply small matrices in a single step, usually at reduced precision.

Around the compute die sit stacks of high-bandwidth memory, HBM, which hold the model’s weights.

To write one token, the chip reads every weight from memory. While the data streams in, the arithmetic units mostly wait.

A 70-billion-parameter model at 16 bits is 140 GB. At 3.35 TB/s, one full read takes about 42 thousandths of a second, which caps a single conversation near 24 tokens a second, even if it fitted on one chip.

Serving many requests together lets each read of the weights do many people’s work. That is why AI services batch requests.

A 70-billion-parameter model at 16 bits needs 140 GB for its weights alone. One GPU holds 80 GB. Try the other sizes below the picture.

So the model is split across GPUs. Each holds a slice of every layer, and they trade results over fast links after each one.

Training also keeps gradients and optimizer state: about 16 bytes per parameter with the common Adam optimizer, before counting the activations.

To learn from more text at once, copies of the model train on different batches, then combine their gradients so every copy takes the same step.

Meta reports that training its largest Llama 3.1 model took 30.84 million GPU hours.

Done on a single GPU, that would take about 3,500 years.

Meta trained it on up to 16,384 GPUs at once. Divided perfectly, that is about eleven weeks; real training also loses time to failures and waiting. Move the slider to see other counts.

Each of those GPUs can draw up to 700 watts: together about 11.5 megawatts, before cooling, networking and the rest of the building.

In the transformer, every token passes through every weight of each feed-forward block. A bigger block means more arithmetic for every token.

A mixture-of-experts layer splits that block into several smaller ones, the experts, and adds a small router. For each token the router scores the experts and sends the token to the best few, blending their answers. Mixtral uses eight experts in each layer and picks two.

Different tokens take different experts; the router learns its choices during training. They are not tidy subjects: in Mixtral, experts did not split by topic, and the patterns found were closer to grammar.

Every expert must still sit in GPU memory, but each token only does the arithmetic of the ones it visits. A model can hold far more than it uses on any one token.

One GPU: a compute die, with memory stacked beside it.

Eight GPUs share a server, joined by fast links, such as NVIDIA’s NVLink, so that they can work almost as one.

Servers fill racks, and racks are wired together by a network that carries results and gradients between thousands of GPUs.

Almost all the electricity a data centre uses ends up as heat. Cold air or liquid flows in, hot air or liquid flows out, around the clock.

Your question arrives as tokens.

The model reads the whole prompt in one pass, every position in parallel. This is the pause before the first word appears.

Then it writes one token per step. Each step reuses what earlier tokens left in the KV cache, so nothing is computed twice, and the cache grows with the conversation.

A server interleaves many conversations and adds new ones as others finish, so every read of the weights serves as many people as it can.

Training usually keeps a 32-bit copy of every weight: precise, and large. For a 70-billion-parameter model, 280 GB.

Most models run in 16 bits. Here the weight moves by less than a ten-thousandth.

At 8 bits, the values a weight can take thin out. This one lands about 0.02 away, and the model halves again.

At 4 bits, sixteen levels times a shared scale stand in for every weight. The model fits in 35 GB, on one GPU, at some cost in quality.

Vision models cut an image into patches: small squares of pixels.

Each patch becomes a vector, like a word, and a transformer reads them together.

Image generators work the other way round. Training teaches a model to remove a little noise from pictures, one step at a time.

To draw, it starts from pure noise and removes it step after step, guided by your words, until a picture is left. The steps shown are one picture’s noised versions, played backwards.

Your question reaches the model.

When it lacks something, the model can write a request for a tool, such as a search, instead of an answer. The software around it decides whether that call is allowed.

The tool’s result is added to the model’s context, the same way your question was.

With the result in view, the model writes its answer, and can point to where it came from.

An answer can read fluently and confidently.

Each token was chosen because it was likely in context. Nothing in that process checks the claim against the world.

A reliable source settles it: sunlight scatters off the gases in the air, and blue light scatters more than red.

Tools and sources help, but judgment stays with the reader. Ask where a claim comes from before you rely on it.