Research Deep Dive · Xiaohu Insights

Edinburgh University Releases a Free 20-Page Paper: The Simple Math Behind ChatGPT

Fed up with AI jargon, a statistician re-explained the large language model from scratch using the standard language of his own field.

The 60-Second Read

A researcher with decades of experience in statistical modeling wanted to understand LLMs but got tripped up by terms like attention, Query, and Key. His conclusion: the math isn't hard—it's the jargon that's the barrier.
So he wrote a free 20-page paper re-explaining the math inside ChatGPT using standard statistical terminology. All you need is three things from your first year of university.
Attention, the most hyped concept, is just a weighted average that a high school student could grasp. The real complexity lies elsewhere.

The Backstory

A Math Expert Couldn't Read AI Papers, and He Blames the Jargon

The Credit Research Centre (CRC) at the University of Edinburgh Business School has published a free 20-page paper titled "The Simple Mathematics of Large Language Models." After reading it, you'll be able to explain exactly what's being computed inside ChatGPT and Claude, without wading through multi-hundred-page deep learning textbooks. All it requires is a working knowledge of matrix multiplication, conditional probability, and the concept that statistical model parameters are estimated from data.

The author is Joseph L. Breeden, who has spent decades working on nonlinear systems modeling, statistics, and linguistics, and runs his own analytics company. He expected reading about LLMs to be straightforward, but the vocabulary tripped him up:

"A token isn't a 'query.' Linguists don't say 'key.' And what in the world is a 'head'? I could guess these words came from computer science, but even the 'no-jargon' explainers start throwing around query, key, value. So let's just start over."
Joseph L. Breeden, "The Simple Mathematics of Large Language Models"

Attention, Query, Key, Value, head—these terms were borrowed from database theory, cognitive science, and electrical engineering. These borrowings are imprecise, and they hide the fact that the math is actually quite ordinary. So he replaced all of them with names that already exist in statistics, explaining along the way what was wrong with the original terms. This renaming effort is the paper's second major thread; we've compiled it into a glossary at the end of this article.

The Problem

An LLM Does One Thing: Guess the Next Word

A language model takes a piece of text and outputs a probability distribution for what the next word might be. Given "The cat sat on the," the words mat, floor, and chair will get high probabilities, while democracy gets a probability close to zero.

Writing an essay just repeats this one action hundreds of times: compute a probability distribution, pick a word, add it to the context, and repeat. You can step through this loop in the demo below.

Click a step, and the model writes one more word 1234

The cat sat on the ?

mat42%

floor18%

chair9%

democracy≈0%

In this step, the model does one thing: assign a probability to every word in its vocabulary. It picks the word with the highest probability—mat.

The cat sat on the mat ?

. (period)37%

and14%

while8%

purple≈0%

The freshly written mat is now part of the context. The same process repeats, but the probability distribution looks completely different this time.

The cat sat on the mat. It ?

was25%

looked12%

purred7%

democracy≈0%

The longer the context, the more constrained the choices become. Knowing that "It" refers to the cat requires the first half of the sentence. How that works is the topic of the next few sections.

The patient was diagnosed with a rare ?

blood disease29%

genetic disorder21%

tumor12%

Tuesday≈0%

Same process, different language and topic. When you read that sentence fragment, you also know a disease name is coming. That's the predictability the model has learned.

Interactive demo (made by us; percentages are illustrative, not measured): Four steps walk through the complete loop—"compute probabilities → pick a word → add it to the context → repeat." Writing a full essay is this loop repeated hundreds or thousands of times.

Why does modeling the probability of a sequence of words suffice? Probability theory gives us an identity called the chain rule (an identity, not an assumption): the probability of an entire sentence can be factored into a product of conditional probabilities—the probability of the first word, times the probability of the second word given the first, times the probability of the third given the first two, and so on. So if you can compute "the next word given the context," you've modeled the entire language. That's all an LLM does.

Where the Difficulty Lies: More Contexts Than Atoms in the Universe

The challenge isn't the formula itself; it's the number of possible contexts. With a vocabulary of, say, 50,000 tokens, a 10-word context has 50,000¹⁰ possible combinations—a number larger than the estimated number of atoms in the observable universe. Looking things up in a table is hopeless. You can't store a probability table for every possible context.

So you have to learn a function that generalizes: if you've seen "the cat sat on the mat," you can handle "the dog lay on the rug." The rest of the paper is about what that function looks like.

Two reality checks. First, calling LLMs "stochastic parrots" doesn't hold up—a parrot stitches phrases from shallow frequency, but an LLM processes substantial context before generating the next word. If the context is "Genie explained she couldn't clean the floor because…", the next word is almost forced. Second, a cold shower the other way: an LLM is, strictly, a finite-order Markov model. The context window is a hard limit; anything outside it is, in principle, invisible to the model, no matter the data.

Representation

Machines Don't Read: Words as Points, Similar Ones Nearby

The lazy approach is to give each token an ID: word 1, word 2, etc. In math, that's a long vector with a single 1 and zeros elsewhere. But that's like declaring all words are unrelated: the distance from "cat" to "dog" equals that from "cat" to "however." Clearly wrong. Cats and dogs are both animals, both pets, and both nouns.

A better approach: represent each word as a point in a high-dimensional space (typically 1,000 to 10,000 dimensions), placing words with similar meanings or grammatical functions close together and unrelated words far apart. The coordinates are learned from data during training; no one sets them by hand. This technique is called an embedding. In statistical terms, it resembles Principal Component Analysis (PCA): compressing many attributes into fewer dimensions. The resulting dimensions may have intuitive meanings, or not.

Illustration (drawn by us): Once words become points in space, "similarity" becomes measurable distance, and "gender difference" becomes a direction you can add or subtract.

This idea has roots in linguistics. In 1957, linguist J.R. Firth observed, "You shall know a word by the company it keeps." Words in similar contexts tend to have similar meanings. Models learn word embeddings precisely by exploiting this rule: words that can be substituted for each other in many sentences end up with similar vectors. Word2vec in 2013 proved this approach works and produced that famous example: king − man + woman ≈ queen.

The Catch

But a Word Has Many Meanings, and One Point Isn't Enough

Once we have our points, a new problem appears. Consider the English word bank:

Sign in