An LLM Does One Thing: Guess the Next Word

A language model takes a piece of text and outputs a probability distribution for what the next word might be. Given "The cat sat on the," the words mat, floor, and chair will get high probabilities, while democracy gets a probability close to zero.

Writing an essay just repeats this one action hundreds of times: compute a probability distribution, pick a word, add it to the context, and repeat. You can step through this loop in the demo below.

Why does modeling the probability of a sequence of words suffice? Probability theory gives us an identity called the chain rule (an identity, not an assumption): the probability of an entire sentence can be factored into a product of conditional probabilities—the probability of the first word, times the probability of the second word given the first, times the probability of the third given the first two, and so on. So if you can compute "the next word given the context," you've modeled the entire language. That's all an LLM does.

Where the Difficulty Lies: More Contexts Than Atoms in the Universe

The challenge isn't the formula itself; it's the number of possible contexts. With a vocabulary of, say, 50,000 tokens, a 10-word context has 50,000¹⁰ possible combinations—a number larger than the estimated number of atoms in the observable universe. Looking things up in a table is hopeless. You can't store a probability table for every possible context.

So you have to learn a function that generalizes: if you've seen "the cat sat on the mat," you can handle "the dog lay on the rug." The rest of the paper is about what that function looks like.

Machines Don't Read: Words as Points, Similar Ones Nearby

The lazy approach is to give each token an ID: word 1, word 2, etc. In math, that's a long vector with a single 1 and zeros elsewhere. But that's like declaring all words are unrelated: the distance from "cat" to "dog" equals that from "cat" to "however." Clearly wrong. Cats and dogs are both animals, both pets, and both nouns.

A better approach: represent each word as a point in a high-dimensional space (typically 1,000 to 10,000 dimensions), placing words with similar meanings or grammatical functions close together and unrelated words far apart. The coordinates are learned from data during training; no one sets them by hand. This technique is called an embedding. In statistical terms, it resembles Principal Component Analysis (PCA): compressing many attributes into fewer dimensions. The resulting dimensions may have intuitive meanings, or not.

This idea has roots in linguistics. In 1957, linguist J.R. Firth observed, "You shall know a word by the company it keeps." Words in similar contexts tend to have similar meanings. Models learn word embeddings precisely by exploiting this rule: words that can be substituted for each other in many sentences end up with similar vectors. Word2vec in 2013 proved this approach works and produced that famous example: king − man + woman ≈ queen.

But a Word Has Many Meanings, and One Point Isn't Enough

Once we have our points, a new problem appears. Consider the English word bank:

What we need is a point that deforms with context: the same "bank" is near "money" in the savings sentence, near "river" in the walking sentence, shaped by the surrounding words. A single fixed point per word isn't enough. The entire transformer architecture exists to perform this shaping.

Attention Sounds Mysterious, But It's a Weighted Average

This is the heart of the paper. The author's demystification of attention boils down to one sentence: mathematically, it's a weighted average. The only clever part is that the weights are learned from data, not set by hand.

Take "The cat sat on the mat." Suppose we want a new, context-aware representation for sat. "cat" is the subject (who is sitting), so its weight should be large. "mat" is the location, so it gets a medium weight. "the" is a function word with little semantic content, so its weight should be small. Use these weights to average the content of each word, and the result is a "sat" that knows about the cat and the mat.

The author makes two renames here. He calls the jargon term attention score the influence score and attention weight the influence weight. His reasoning: the latter names describe what the number actually does in the computation, while the former are just anthropomorphic metaphors.

How to Compute the Weights: The Dot Product

The more aligned two vectors are, the larger their dot product. If they point in opposite directions, the dot product is negative. The dot product was chosen for two reasons: it's the simplest of all similarity measures and fast to compute; and it's differentiable, which is required for gradient descent learning.

Why Not Just Dot Product the Word Vectors Directly? Why the Three Matrices?

In the jargon, these three matrices are called Query, Key, and Value. The author rejects this database metaphor and uses neutral labels W^A, W^B, and W^C. Position i isn't "querying" anything; there's no database and nothing is being "looked up."

Passing each word vector through its own linear transformation before taking the dot product solves two specific problems:

The paper's own example clarifies the second point. Still computing the representation for "sat":

"I have a grammatical relation to this verb" and "I am a cat" are two different kinds of information. Splitting them into two matrices lets the model learn them separately. What's in these matrices initially? Random numbers that extract nothing. It's only through training on trillions of tokens, with the parameters repeatedly adjusted, that they become useful feature extractors. These structures are found, not designed.

From Scores to Weights: The Real Name for Softmax

Dot products can be positive or negative, so they can't be used directly as weights. They are first divided by √k (where k is the vector dimension) to prevent numerical issues, then passed through an exponential normalization, which turns a list of real numbers into a set of non-negative weights that sum to exactly 1.

Two further notes for the mathematically inclined. First, the entire influence score is a bilinear form. Factoring it into two smaller matrices multiplied together is called low-rank parameterization, which reduces the number of parameters and acts as a form of regularization against overfitting. Second, these matrices are not uniquely determined: multiplying them by an orthogonal matrix leaves the predictions completely unchanged. So don't expect to decompose one of these matrices and "read off" what it has learned. This is the same old problem as in factor analysis, where factors are only determined up to a rotation.

"Multi-Head" Isn't Mysterious: Dozens of Sets Learn in Parallel, and the Division of Labor Emerges on Its Own

One set of W^A/W^B/W^C can learn only one pattern of influence. But words relate to each other in more than one way: subject-verb is a syntactic relation, near-synonyms have a semantic relation, a pronoun referring back to a previous noun is an anaphoric relation, and adjacent words often have grammatical ties.

The solution: run H sets of matrices in parallel, let each learn its own thing, then concatenate the outputs and project back to the original dimension. In practice, H is typically between 12 and 96.

The jargon calls each set an attention head. This term draws the paper's strongest criticism. "Head" suggests a designed, specialized component, like the read/write head of a hard drive, each responsible for reading specific data. The reality is the opposite,

This has a direct analogue in statistics. In factor analysis, factors differentiate for the same reason: it's more efficient to explain different sources of variance separately. No one assigns them roles. So the author suggests calling these parallel sets "relations," acknowledging that the original architectural inspiration came from linguistics while also acknowledging that the learned patterns only partially match linguistic categories.

Stack Dozens of Layers, Remember Word Order, and the Machine Is Complete

One pass of weighted averaging is only enough for words to absorb nearby information. But some dependencies span a long distance. Consider this sentence:

"The scientist who discovered high-temperature superconductors won the Nobel ____ Prize."

The blank is almost certainly "Physics," but to figure that out, you have to connect "Nobel Prize" all the way back across the long relative clause to "superconductors" and "scientist." The solution is to stack many layers of this weighted-average operation, feeding the output of one layer into the next. Modern LLMs stack 32 to 128 layers, each with its own complete set of parameters, sharing nothing with the other layers. This is where the explosion in parameter counts comes from: a model with a dimension of 12,288, 96 layers, and 96 relations per layer has hundreds of billions of numbers in these matrices alone.

The Three Additions in Each Layer, Each Fixing One Problem

The nonlinearity prevents "wasted stacking." The most common choice is ReLU, which does exactly one thing: turns all negative numbers into zero. Without it, multiple layers of linear transformations collapse into a single layer (linear composed with linear is still linear), so stacking layers has no effect.

The residual connection prevents training signals from failing to propagate. Instead of recomputing the entire representation in each layer, it makes an incremental correction: output = input + correction. During training, the prediction error must be backpropagated from the last layer to the first. The chain of multiplied derivatives shrinks exponentially, meaning early layers get no signal—this is the vanishing gradient problem. The additive structure ensures there's always an identity path in the derivative, so the signal has a route back.

Normalization prevents numerical values from spiraling out of control. After each step, the vector is scaled to have mean 0 and variance 1, so numbers don't grow or shrink unstably across dozens of layers.

One More Thing: This Machine Is Naturally Blind to Order

There's a pitfall we skipped. The weighted-average operation treats the context as a bag of words, not a sequence. Shuffle the words in a sentence, and the computed influence scores remain unchanged. But "dog bites man" and "man bites dog" are clearly not the same thing.

So position has to be fed in explicitly. An early approach was to add a "position vector" to each word embedding, marking it as the nth word. The problem is extrapolation: if the model never saw the 10,000th position during training, its vector is effectively untrained, and performance degrades on long texts.

Linguistic intuition, however, says what matters is relative position—whether a verb is two words away from its subject is the same whether they're at positions 5 and 3 or 500 and 498. The now-dominant RoPE (Rotary Position Embedding) follows this logic: instead of adding a vector, it rotates the vector by an angle proportional to its position. When computing the dot product, the two angles automatically subtract, leaving only the distance between the two words. The alternative, ALiBi, is simpler: the influence score is penalized linearly with distance, leaving the word vectors themselves completely unchanged.

Training Is Simple: When You Guess Wrong, Adjust Until You Guess Right

With the structure in place, how are those hundreds of billions of numbers determined? Let's look at the output side: the final layer's representation is dot-producted with the output vector of every word in the vocabulary, and the result is passed through that same softmax from before. That yields the probability distribution for the next word. The author points out a useful fact: this final step is a multinomial logistic regression with 50,000 classes, using the final representation as input features.

The training objective is maximum likelihood, the standard recipe in statistics: adjust the parameters to maximize the probability the model assigns to the sentences that actually occur in the training corpus. Taking the logarithm turns the product into a sum, which is equivalent to minimizing the cross-entropy.

How do you minimize it? Gradient descent. The entire model, from embeddings to output probabilities, is a composition of differentiable functions. The chain rule of calculus gives you the derivative of the objective with respect to any parameter, and the rest is handled by software. In practice, training uses stochastic gradient descent: the derivative is estimated from a small batch of data at each step. The estimate is noisy but unbiased, and convergence still happens.

The author cautions against thinking of training as "one giant stew"—that's a misconception. Real frontier model training happens in several distinct stages, each carefully designed:

His analogy is human education: children go to school in stages, building foundations before tackling advanced material. No one is immersed in all knowledge at once. LLM training has converged on the same idea.

What you get is a model that has read more text than any person could in a lifetime. And considering how much writing has been produced and then never read by anyone, it may have read more than all of humanity combined.

The Model Learns a Great Deal, But Four Things Are Structurally Impossible

Dissecting a trained model reveals that structures no one wrote in have emerged on their own. Some relation groups track syntax, others track pronouns back to their referents. Shallow layers tend to handle local grammar and parts of speech, while deeper layers move toward semantics, discourse structure, and long-range relationships. At sufficient scale, a qualitative change occurs—from the GPT-3 generation with 175 billion parameters onward, coherent writing, reasoning-like question answering, factual knowledge, translation, summarization, and code generation all emerge.

The author places this in the broad family of "emergence": thermodynamics emerges from statistical mechanics, fluid dynamics from molecular interactions, and the complex choreography of bird flocks from a few simple rules each bird follows. Macroscopic phenomena arise from microscopic mechanisms, and you can't see it by staring at the low-level rules. Why a goal as simple as "predict the next word" grows capabilities that no one explicitly wrote in, at sufficient scale, remains an open scientific question.

The other half has to be said: four things are structurally impossible for this mechanism.

The author's suggested direction is to use LLMs as dispatchers: let the model translate a human intention into a well-defined problem, hand it off to a specialized solver (a theorem prover, a symbolic math system, a database), and translate the result back into human language. Such hybrid architectures are emerging, but they are not yet as mature as the monolithic approach.

What to Take Away: A Jargon Translation Table, and the 20-Page Paper Itself

Here's the entire translation effort of the paper condensed into a table. The left column is the term you'll see elsewhere. The middle is its mathematical name. The last column is what was wrong with the original term.

The paper's conclusion in a few sentences: an LLM is a conditional probability distribution "next word given context." The core mechanism is a learned weighted average. Influence is measured by a dot product of projected vectors. Stacking many layers, each with many parallel relation groups, allows learning multiple levels of complex relationships. The mathematics involved goes no deeper than linear algebra (matrix multiplication, dot products), probability (conditional distributions, exponential normalization), and optimization (gradient descent). The truly remarkable part is the combination and the scale: when these ordinary components are assembled this way and scaled to this magnitude, capabilities emerge that look like forming concepts and recognizing complex relationships.

An LLM Does One Thing: Guess the Next Word

Given a piece of text, the model computes probabilities for the next word. For "The cat sat on the___," "mat" gets a high probability; "democracy" gets almost zero. By the chain rule of probability—an identity, not an assumption—the probability of an entire sentence factors into a product of conditional probabilities. The model's entire job is computing that product. The difficulty is combinatorial explosion: with a vocabulary of 50,000 tokens, a 10-word context has 50,000¹⁰ possibilities—more than the atoms in the observable universe. A lookup table is impossible. You have to learn a function that generalizes.

Attention—the Most Hyped Concept—Is Just a Weighted Average

Words must first be turned into points in a high-dimensional space, with similar words close together and coordinates learned from data. But a single word can have multiple meanings: "bank" in "savings" is different from "bank" in "river." A fixed point can't capture that. What you need is a point that moves based on context. That's the paper's biggest demystification: the mechanism responsible for moving the points, called attention, is mathematically just a weighted average. It's like calculating a final grade: homework counts 30%, final exam 70%. Here, the weights are learned from data.

How to Assemble the Whole: Run Dozens in Parallel, Then Stack Dozens of Layers

One set of matrices can learn only one kind of relation. But words relate in many ways: syntax, semantics, anaphora, proximity. So you run H sets in parallel, letting each learn its own pattern. The jargon for these is "attention heads," and that's the term the paper criticizes most: it implies each head has a pre-assigned role. In reality, all H sets start from random and are trained on the same objective; the division of labor emerges on its own. One pass of the weighted average only lets words see their neighbors. Long-range dependencies require stacking many layers on top.

The Model Learns a Lot, But Four Things Are Structurally Impossible

In a trained model, you can see structures that no one wrote in: some relation groups track syntax, others track anaphora. At sufficient scale, capabilities change qualitatively—from coherent writing to translation, summarization, and code generation. Why this happens is still an open scientific question. But there are four hard limits, listed in the paper.