Research explainer · XiaoHu reads the paper

The model you paid for may not be the one answering you — and one random number can prove it

Sample the same model twice and the median fingerprint distance is 0.140. One endpoint marketed as a proprietary in-house flagship sits 0.141 away from open-source Qwen. Which is to say: it is indistinguishable from open-source Qwen.

The one-minute version

When you buy AI through a reseller or an aggregator, you pick a name and you get back a block of text. Nothing in between proves the answer came from the model you selected. One earlier study probed 31 commercial endpoints serving Llama; 11 produced outputs that didn't match the released weights.
The test is a single sentence: ask it over and over for "a random number between 1 and 100." It keeps landing on the same handful of numbers, and every model plays favorites differently. That table of answers is the fingerprint, and each query costs exactly one output token.
Why it's hard to dodge: benchmarks measure whether a model gets the answer right; the fingerprint measures how it splits probability across several answers that are all equally fine. Quietly swapping in a compressed build disturbs exactly that layer.
Accuracy and price: 40 questions per endpoint gives a 7.3% error rate — roughly one wrong call in 14. Just 8 questions (about 120 queries) still gets you 10.6%. An audit costs a few cents, so you can re-run it on the same endpoint daily.
The headline finding: Tomáš Bruckner of the Prague University of Economics and Business tested all 165 models for a total of $34.44. One endpoint sold as a proprietary in-house flagship (writer/palmyra-x5) fingerprints indistinguishably from open-source Qwen — the numbers are in the dek, the details in the "What the survey found" section.

The problem

You pay for a name, not for a machine

You pay for a top-tier model; the thing answering you might be something much cheaper. Aggregators like OpenRouter are genuinely convenient — hundreds of models behind one endpoint. But all you select is a name. Which machine your request lands on, and what's running there, is invisible to you, and nothing certifies it.

Two layers of handoff sit between you and the machine doing the work; the aggregator itself acknowledges that quantization levels vary across upstream providers. The "11 of 31 endpoints" figure comes from prior work cited in the paper

← swipe to see the full figure →

Checking is genuinely hard. From the outside you can't get the weights (the big bundle of parameters that is the model itself), and you can't get the logitsthe raw scores the model assigns to each candidate answer. Most production endpoints hand back only the final text and withhold these entirely either. Watermarking and backdoors require planting a mark in the model ahead of time, so only the model's owner can use them. And you need to probe dozens of endpoints repeatedly, cheaply enough to keep doing it.

Where the existing approaches stall

Approach	What it needs	Where it stalls
Text attribution classifiers	full long-form generations	Can trace a whole chat reply back to its source model with 97% accuracy across five candidates, but the classifier has to consume long outputs, which drives up the cost of a single audit
Purpose-built probes LLMmap / TRAP	hand-designed or adversarially optimized prompts	LLMmap identifies 42 model versions from eight queries, but the distinctiveness of those prompts is itself the weakness: a provider spots them instantly and routes just those to the real model
Rank-based statistical tests uniformity tests over ranks	log probabilities returned by the model	Built for quantization detection, but plenty of endpoints return text only, so the premise never holds
Statistical equivalence testing	full sampled strings, plus a reference deployment of the same model for calibration	Closest to this paper's setting, but it burns full generations and calibrates against only a single reference deployment
Single-token answer distributions this paper	one output token — no weights, no logits, no cooperation from the model owner	The trade-off: a single query proves nothing. You need dozens to hundreds of repeats before a distribution emerges

← swipe to see the full table →

LLMmap and its cousins obsess over what special prompt to send. This paper inverts that: the questions are as ordinary and harmless as possible, endlessly rephrasable, and what matters is how the model answers. Identity hides in the distribution.

Method and mechanism

Ask for a random number, and the spread of answers gives it away

Tomáš Bruckner at the Prague University of Economics and Business found a neat trick: ask an AI for a random number, and how it answers is enough to identify it.

The trick exploits a famously bad habit: models can't produce real randomness. Ask for a number between 1 and 100 and the same one or two keep coming back — and every model has its own pet numbers. GPT-4o leans hard on 42 (with 37 and 57 close behind), Claude Sonnet 5 says almost nothing but 47, Llama 3.3 locks onto 53, and Qwen3-Max answered 42 on all 30 samples. Everyone else has treated this as a defect to fix. This paper treats it as a fingerprint.

The core recipe

Ask for "a random number between 1 and 100," demand a one-word reply (16-token cap), turn off reasoning mode, set temperature to 1.0 — which just means let it sample from its own probabilities instead of always grabbing the safest answer — and repeat a few dozen times. Whether the answer comes back as "forty-two," "٤٢," or "42," it counts as 42. Tally how often each answer appears and you have a distribution.

That distribution is the fingerprint. Each query burns exactly one output token.

Here's what four of them look like:

Answer distributions from four models on the 1-to-100 random number question

One question, four answer distributions (English, temperature 1.0, 30 samples each). GPT-4o spreads its probability over 42, 37 and 57, with 42 peaking near 0.44; Claude Sonnet 5 piles almost everything on 47; Llama 3.3 on 53; Qwen3-Max answers 42 every single time. Source: arXiv:2607.10252 Fig.1

The lopsidedness is measurable. Slice 165 models × 40 questions into cells; 6,572 cells had at least 10 valid answers. Across those, the median entropyhow hard the answer is to guess. About 1 bit means the answer is essentially bouncing between two options of the answers is just 1.00 bit, against the 6.64 bits true randomness would give (for the 1-to-100 question). In each cell the model's favorite answer — not only on number questions, but on random words, colors and cities too — typically takes about seven in ten replies (median share 0.71).

How big that gap is

At 6.64 bits, every number from 1 to 100 is genuinely in play and you have no idea which one is coming. At 1.00 bit, the model is really just toggling between two numbers, and your odds of guessing right are close to even. It agreed to be random; what it delivered was its verbal tic.

One set of weights, several names in the catalog

Before any of this works, you have to be clear about what's being measured. In an aggregator's catalog, names map to weights many-to-one: the same weights sit there under a rolling -latest alias, a dated snapshot, a free tier, a thinking tier, a fast tier, a long-context tier.

So the paper distinguishes three levels. A checkpoint is one specific set of weights, and that's what the fingerprint tracks. An endpoint is a service path that claims to lead to some checkpoint. A family is a lineage of checkpoints descended from a common ancestor. Verification asks whether an endpoint serves the weights it claims; attribution asks where those weights came from. The name you bought guarantees neither.

Why one token carries identity at all

Four layers stack up inside that distribution, and each writes its own trace into it.