When a senior engineer says “I’m pretty sure this is right,” you hear useful information. When they say it with the same tone about something that turns out to be completely wrong, you update your trust accordingly. Language models have no equivalent mechanism. The confidence signal is flatlined at maximum, all the time, for everything.
This isn’t a bug that gets patched in the next release. It’s structural. Understanding why requires looking at how these systems are actually built, and what that means for the people relying on them.
1. Confidence Is Not a Model Output, It’s a Style Choice
Large language models don’t output a confidence score alongside their text. They output tokens, one at a time, each predicted based on what’s statistically likely to follow the previous ones. The certainty you hear in the phrasing, “The capital of France is Paris” versus “I believe the capital might be…” isn’t drawn from some internal certainty meter. It’s a learned stylistic pattern.
Models trained on human-generated text learn that authoritative-sounding text tends to be the kind of text that gets written. Encyclopedias, textbooks, and documentation all sound confident. So the model learns: confident-sounding text is what high-quality output looks like. The result is a system that sounds equally sure when it’s recalling a well-documented fact and when it’s fabricating a citation that doesn’t exist.
2. Token Probabilities Exist, But They Don’t Mean What You Think
There is a number that lives inside these models during inference: the probability assigned to each candidate token at each generation step. If you have API access, you can sometimes retrieve these logprobs. A high probability on a token means the model is, in some sense, “more confident” about that word choice.
But this is deceptive. A model can assign high probability to a confidently-stated falsehood. The probability reflects how likely that token is given everything that came before, not how factually grounded the claim is. If the model has drifted into a plausible-sounding but wrong narrative, each subsequent token can be highly probable within that wrong narrative. The math is internally consistent. The facts are not. Token certainty and factual accuracy are orthogonal dimensions.
3. Calibration Is the Property You Actually Want, and Most Models Lack It
In machine learning, calibration refers to whether a model’s confidence scores actually correlate with its accuracy. A well-calibrated model that says it’s 70% confident should be right about 70% of the time across a large set of such predictions. Classic classifiers and well-tuned probabilistic models can be calibrated. Most large language models, in their default deployment, are not calibrated in any useful sense for factual claims.
This matters enormously in practice. A miscalibrated model that’s wrong 40% of the time but sounds right 100% of the time is more dangerous than a clearly uncertain system you know not to trust blindly. At least a system that hedges gives you a signal to act on. The uniform-confidence model forces all the epistemic work onto the user, who now has to develop independent judgment about every single output, which largely defeats the efficiency argument for using the tool in the first place.
4. RLHF Made This Worse in a Specific Way
Reinforcement Learning from Human Feedback (RLHF) is the training technique that made models like ChatGPT feel more natural and helpful to interact with. Human raters evaluate model outputs and the model is tuned to produce outputs raters prefer. This is genuinely useful for many things. It is bad for epistemic calibration.
Human raters, it turns out, tend to prefer confident-sounding answers. A response that says “I’m not certain, but the answer may be around X” often rates lower than one that states X directly and fluently. So RLHF systematically selects against hedging. The model learns that confidence reads as quality. This is one of the reasons that as models have gotten better at following instructions and sounding natural, hallucinations haven’t disappeared the way you’d naively expect from capability improvements alone. The relationship between model sophistication and confident-sounding errors is real.
5. The Problem Compounds When Models Write Code
Text that sounds wrong can at least be caught by a reader with domain knowledge. Code that sounds confident and compiles successfully is harder to audit. A language model can write a function that looks syntactically correct, passes a superficial read, and fails in production for a reason that only surfaces under specific conditions. The model wrote it with the same stylistic assurance it writes everything else.
This is worth sitting with: the model has no feedback loop from execution. It has never run the code it writes. The confidence in the code comments, the clean variable names, the confident return statements, all of that is style. The actual correctness is something you have to verify externally. Using AI-generated code without tests is trusting a confident-sounding stranger’s directions without a map.
6. Some Architectures Are Trying to Fix This
Retrieval-augmented generation (RAG) is one partial solution. Instead of relying on what’s baked into the model weights, the system retrieves relevant documents at query time and grounds the response in them. This doesn’t solve the confidence calibration problem, but it does give you a source to check. When the model says something and you can see the retrieved passage it drew from, you have a verification handle.
More interesting are approaches where models are explicitly trained to express uncertainty, to say “I don’t know” or “I don’t have reliable information about this.” OpenAI, Anthropic, and others have done work in this direction. The results are genuinely better in some domains, but the core tension remains: models trained on human preferences will continue to face pressure toward confident-sounding outputs because that’s what users respond to positively in the short term, even if it misleads them.
7. The Practical Takeaway Is About Trust Architecture, Not Individual Outputs
The right response to uniform model confidence isn’t to distrust everything equally, which would make the tools useless. It’s to build a mental model of where these systems tend to go wrong and weight your verification effort accordingly. Models are generally reliable on well-documented, frequently-occurring facts. They’re unreliable on specific numbers, dates, citations, recent events, and anything that requires precise recall of a particular source.
Treat the output like a first draft from a very fast, very well-read colleague who occasionally makes things up without realizing it and shows no outward sign when they do. You wouldn’t publish that colleague’s draft without reading it. You’d read it more carefully on the sections that require precision. The confidence in the prose tells you nothing about where to focus that attention. You have to bring your own calibration to the interaction, which is the uncomfortable but honest conclusion here.
