Ask for a random number, and the spread of answers gives it away

Tomáš Bruckner at the Prague University of Economics and Business found a neat trick: ask an AI for a random number, and how it answers is enough to identify it.

The trick exploits a famously bad habit: models can't produce real randomness. Ask for a number between 1 and 100 and the same one or two keep coming back — and every model has its own pet numbers. GPT-4o leans hard on 42 (with 37 and 57 close behind), Claude Sonnet 5 says almost nothing but 47, Llama 3.3 locks onto 53, and Qwen3-Max answered 42 on all 30 samples. Everyone else has treated this as a defect to fix. This paper treats it as a fingerprint.

Here's what four of them look like:

One question, four answer distributions (English, temperature 1.0, 30 samples each). GPT-4o spreads its probability over 42, 37 and 57, with 42 peaking near 0.44; Claude Sonnet 5 piles almost everything on 47; Llama 3.3 on 53; Qwen3-Max answers 42 every single time. Source: arXiv:2607.10252 Fig.1

The lopsidedness is measurable. Slice 165 models × 40 questions into cells; 6,572 cells had at least 10 valid answers. Across those, the median entropyhow hard the answer is to guess. About 1 bit means the answer is essentially bouncing between two options of the answers is just 1.00 bit, against the 6.64 bits true randomness would give (for the 1-to-100 question). In each cell the model's favorite answer — not only on number questions, but on random words, colors and cities too — typically takes about seven in ten replies (median share 0.71).

One set of weights, several names in the catalog

Before any of this works, you have to be clear about what's being measured. In an aggregator's catalog, names map to weights many-to-one: the same weights sit there under a rolling -latest alias, a dated snapshot, a free tier, a thinking tier, a fast tier, a long-context tier.

So the paper distinguishes three levels. A checkpoint is one specific set of weights, and that's what the fingerprint tracks. An endpoint is a service path that claims to lead to some checkpoint. A family is a lineage of checkpoints descended from a common ancestor. Verification asks whether an endpoint serves the weights it claims; attribution asks where those weights came from. The name you bought guarantees neither.

Why one token carries identity at all

Four layers stack up inside that distribution, and each writes its own trace into it.

At this point you might ask: if temperature 0 is deterministic on 90.4% of questions within a provider, why not just compare temperature-0 answers and be done in three queries? Three reasons. Safe answers collide across vendors (if everyone says 42, nobody is distinguishable). Cross-provider determinism falls to 84.5%, so a single point is unstable on its own. And a full distribution carries far more information than one point. Hence the need to make it "random" and ask repeatedly.

Why quantization and distillation can't hide from this question

Quantization rounds weights to lower precision; distillation trains a small model with a big one as teacher. Take quantization first. On questions the model is confident about, that rounding error doesn't change which answer wins, which is why benchmarks barely register it. But when several candidates are neck and neck, the error is plenty to scramble their ordering.

And "give me a random number" is precisely that kind of question: several pet numbers are all defensible, no ranking among them is right or wrong, and the whole thing rides on a delicate split of probability. That split is what the fingerprint reads.

Turning "looks similar" into a number you can compare

One question isn't safe: two models could easily both love 42. So the battery is 10 questions, asked in 4 languages each — 40 in total.

Why ten questions, and why four languages

The paper gives one sentence of rationale for each of these three design choices; what follows unpacks them.

Closed and open questions do different jobs. A coin flip has two answers, 1-to-10 has ten, and every model is choosing from the same menu — what's compared is the fine-grained split. But two options can only differ so much; the ceiling is low. Open questions (random word, city, animal) have unbounded answer spaces, so two models might not overlap on a single answer and the distance pins to the maximum. One type teases apart close relatives; the other shoves unrelated models far apart.

"Random" and "favorite" are two different questions. Asking for a random number treats the model as a lottery machine; asking for its favorite number probes its persona, and persona is sculpted by post-training. Two models might collide on random numbers and diverge on favorites, so the pairing buys a free extra dimension.

Ask in another language and you tap a different pocket of memory. The common answer to "name a city" in Chinese-language data isn't the common answer in English. Four languages means four independent slices of signal. Turn that around: anyone trying to impersonate has to match all four sets of habits.

Collapsing two distributions into one distance

The gap between two distributions gets folded into a 0-to-1 number via Jensen-Shannon divergencea measure of how far apart two answer distributions are, from 0 to 1. Symmetric, and still well-defined when the two tables share no answers at all. There's a practical reason for that choice: even when two models name entirely non-overlapping sets of cities, it still yields a meaningful distance — exactly what open questions require. The distance between two models is then the average across questions where both have enough samples.

At the level of a single question, a model compared against itself scores 0.075, and two different models score 0.489. (Mind the units: those are per-question numbers, not directly comparable to the 40-question averages below.) Averaged, they become the scale.

Does the fingerprint survive a change of provider?

One make-or-break question comes first: does the fingerprint change when the provider does? If it does, you're measuring data centers, not models. OpenRouter supplied a natural experiment: of the 165 models, 31 were served by two or more providers with enough samples, yielding 34 directly comparable pairs with at least 15 shared questions each.

Verdict: the fingerprint largely survives a change of provider. Ten pairs didn't; the two most striking get their own treatment in "What the survey found."

What one identity check costs

Checking whether an endpoint is serving the real thing takes four steps.

Scoring works like a fingerprint lock. "Genuine" trials compare two halves of the same model's samples; "impostor" trials compare half of model A against half of model B. Across 165 genuine and 27,060 impostor trials, the full 40-question battery gets AUC 0.971 and an equal error ratethe error rate at the threshold where false accusations and missed substitutions are equally common. 7.3% is roughly one wrong call in 14 of 7.3%.

The verification ROC curve. How to read it: the closer the curve hugs the top-left corner, the better; the diagonal represents random guessing. This one runs almost straight up the left wall, for an area under the curve of 0.971. Source: arXiv:2607.10252 Fig.3

How much accuracy you lose with fewer questions

Equal error rate versus number of questions; the vertical bars show the spread from random subset selection. The curve flattens out after 16 to 24 questions — beyond that, more questions buy little accuracy. Source: arXiv:2607.10252 Fig.4

Even one question beats coin-flipping (50%) by a wide margin. At 16 questions and 15 repeats each, an audit is 240 queries and roughly 16,000 input tokens — under a cent on ordinarily priced models, a few cents on flagships.

The fingerprint can also hint at where a model came from

Same-model fingerprints cluster tightly — what about same-family ones? Compute all pairwise distances across the 165 models, let an algorithm group the close ones (clustering), and the answer is: also close, but less reliably so. This is the weaker set of results in the paper; its main value is as context for the cases in the next section.

165 models clustered by mean fingerprint distance, with leaf labels colored by documented family. Wherever a large block of one color holds together, that's the fingerprint recognizing a family. Source: arXiv:2607.10252 Fig.2

The algorithm is leave-one-out: hide each model in turn, look at whose family its nearest neighbor belongs to, and guess that. Across the 163 models with at least one same-family sibling, it guesses right 59.5% of the time against an 18.4% random baseline — 3.2× better.

The weakness is right there too: ask the tree to split cleanly into 19 well-separated families and it can't. That's mostly not the fingerprint's fault. The labels include an "other" bucket holding 17 unrelated vendors' models, plus a pile of singleton families that can't cluster by construction — the paper says as much, and expects it. Nearest-neighbor lookup recovers lineage; global partitioning doesn't.

Precision is "of those assigned to this family, how many really belong"; recall is "of this family's members, how many were found." GLM's precision of 1.00 means every GLM call was correct. The Llama row deserves a second look: precision 0.88 is respectable, but recall of just 0.58 means nearly half the family wasn't found on home turf. One of the missing shows up in the next section.

Also, 59.5% is a floor. The ground truth comes from published family labels, so when documentation is wrong the fingerprint loses points even if it found the right relative — and errors in the opposite direction can't happen. True performance is no worse than 59.5%.

What turned up when the scale met the real ecosystem

Put the ruler against the live ecosystem and several endpoints come back mismatched. To be clear up front: these are statistical mismatches, not accusations of fraud. Updated weights, licensed quantization and caching layers are all innocent explanations.

Some endpoints don't fingerprint like what they claim to be

Innocent explanations really do exist — training your own model on a Qwen base is entirely legitimate. There's a clean example right here.

deepcogito/cogito-v2.1-671b states plainly that it was trained from a DeepSeek V3 base, and its three nearest fingerprint neighbors are all DeepSeek checkpoints (0.268 to 0.308). Fingerprint and documented lineage agree. xiaomi/mimo-v2.5-pro and inclusionai/ling-2.6-1t likewise sit squarely in Qwen's neighborhood.

Put those two sets of numbers side by side and the distinction sharpens: a documented, legitimate derivative sits around 0.27 from its base — relative distance. Palmyra's 0.141 hugs the 0.140 noise floor — same-model distance. One is well clear of the floor; the other is flat on it. The paper declines to accuse anyone. The numbers are here; read them how you like.

There's also the case of erased lineage. nvidia/llama-3.3-nemotron-super-49b-v1.5 is documented as a Llama-3.3 derivative, yet its fingerprint sits closer to Qwen (0.303) than to any Llama. Post-training went deep enough to bury its origin. That's one source of the 0.58 Llama recall in the previous section, and the ceiling on fingerprint-based attribution.

Same model, two providers, farther apart than two different models

Of the 34 "same model, two providers" pairs, 10 (29%) had already fallen into the "two different models" range. The same model out of two data centers can look like a model swap.

The second row is worth a beat on its own: one vendor, one model name, two paths that both count as official (Azure and OpenAI first-party), and the distance is closing in on the median for two different models. As for how the first row reaches 0.716, the paper draws no conclusion — though the aggregator has itself acknowledged that quantization levels vary across upstream providers.

Three other oddities turned up along the way

Some of this is invisible in the text of a reply and only shows up in the token-by-token ledger. These three aren't trivia; they directly determined which endpoints got dropped from the sample.

The first became an exclusion criterion outright: since there's no way to prove those dozens of tokens aren't hidden reasoning, every such endpoint had to go. Judgments rest on measured behavior, not on an endpoint's self-description — self-descriptions have already been shown to be unreliable.

None of this went through coordinated disclosure. The author's reasoning: these aren't security holes awaiting a patch, and publishing them hands nobody a new attack. Accountability instead comes from releasing all the raw data — every request's timestamp, provider and generation ID is in there, so anyone named, and anyone watching, can re-check the work or push back on it.

What happens once providers start fighting back

What if providers wise up and start countering? Their options come in three tiers, and the method works differently against each.

The defense against T2 is that there's no fixed password in the battery. "Name a random color" and "name a random city" are categories of question, not fixed strings: infinitely rephrasable, infinitely translatable (four languages already, for a start), drawn at random at audit time, and mixable into ordinary traffic.

To filter audit traffic out, a provider would have to catch every single "just name something" question in a flood of normal messages and route it to the real model. Filter too tightly and audit traffic slips through and catches you anyway; filter too loosely and you're handing a big slice of everyday traffic to the real model, burning the savings. Both ends lose. The paper's phrasing is that this is "expensive to detect"; how expensive isn't measured. One gap remains open too: whether the fingerprint holds up when the same kind of question is rephrased was left to future work.

The T3 math is already on its card: mimicry that good is just honestly running the real model, so there's nothing left to save. This method therefore offers no mathematical proof; what it does is make "swap and get away with it" a losing trade — which pairs neatly with the hardware-attestation route (having the chip cryptographically vouch that the weights it ran are the ones claimed).

Does asking the same question repeatedly hit a cache and wreck the result?

Repeating one prompt is bound to hit cache: 13.4% of requests hit the prompt cache, covering 10.1% of input tokens. Does that corrupt the measurement? Depends which kind of cache.

So the author went looking for response caching, with two criteria: answers collapsed to a single value, plus abnormally low and unusually stable latency. There were 2,040 questions where temperature 1.0 produced only one answer (concentrated distributions are common enough on their own), and of those, only 14 also had latency below half the usual. All 14 remained within a normal sub-second range — more like provider load variation than the instant replay a real cache produces. This round turned up no response caching.

Six things this method can't do

One practical note to close on: none of this has to be rebuilt from scratch. The prompts themselves (config/prompts.json) and the raw responses with full service metadata are in the public data package, and the complete collection and analysis pipeline (Node.js and R) is archived separately. Both live behind the Zenodo links in the endnotes. Grab them and run the four steps from "The math" against whatever endpoint you're using.

Model identity leaks through the cheapest possible channel: the single token a model emits when asked a boring question. One Token Is Enough, conclusion

AI has a famous bad habit: it can't produce real randomness. Ask it to pick between 1 and 100 and the same one or two keep coming back — and every model plays favorites differently. Everyone else treated that as a defect to fix. This paper treats it as a fingerprint.

Tally how often each answer appears and you have a distribution, and that distribution is the identity. But "looks similar" is still just a feeling. To put a threshold on it, you have to fold it into a number.

The paper collapses the gap between two distributions into a number from 0 to 1 (Jensen-Shannon divergence, whose virtue is that it still yields a distance when the two sides share no answers at all). Every judgment happens on that scale.

The full 40-question battery errs 7.3% of the time, roughly once in 14; even 8 questions gets you 10.6%. One run settles nothing, but it's cheap enough to re-run on the same endpoint daily. How cheap, measured against the same job: