AI coding assistants are genuinely useful. I use them. Most developers I respect use them. And precisely because of that, we need to be clear about what they fundamentally cannot do: know whether the code they write actually works.
This isn’t a complaint about current quality that will be resolved in the next model release. It’s a structural property of how these systems are built, and misunderstanding it is costing teams real time and real bugs.
The Model Predicts Tokens, Not Behavior
An LLM (large language model) generates code by predicting what text should come next, given the text that came before. It is trained on vast amounts of code, so it has internalized patterns: what a correct-looking SQL query looks like, how you typically structure error handling in Go, what a React component usually contains. It is extraordinarily good at producing text that resembles working code.
But resemblance is not equivalence. The model has no runtime. It cannot execute the function it just wrote. It cannot observe that the off-by-one error on line 14 causes an index out of bounds exception when the input array is empty. It cannot know that the API endpoint it’s calling changed its response schema three months ago. It produces output that is statistically consistent with working code. That is a very different thing from code that works.
To understand what’s actually happening inside these models when they generate your code, the mechanics of context processing matter more than most developers realize. The model is doing something more like very sophisticated pattern matching than reasoning about program semantics.
Confidence Tracks Style, Not Correctness
One of the more insidious properties of code-generating models is that their confidence, expressed through fluency and certainty of output, correlates with how much similar code appeared in training data. Not with whether the code is correct.
The model will produce a clean, well-commented implementation of a subtle concurrency bug with exactly the same apparent confidence it brings to a correct implementation. It produces authoritative-looking nonsense with the same voice it uses for correct solutions. There’s no internal signal you can read that tells you which one you’re looking at. This is a known property of language models more broadly, and it gets more pronounced, not less, as the models get more capable. Confident wrongness scales with model sophistication.
The Verification Problem Is Entirely Yours
Here’s where I want to push back on the way AI coding tools are often marketed: the phrase “it’s like having a senior developer pair programming with you” is doing a lot of misleading work. A senior developer, when writing code, maintains a mental model of execution. They trace through the logic. They think about edge cases because they’ve been burned by those exact edge cases before. They notice when something smells wrong before the tests run.
The AI has none of that. It has no persistent memory of what burned it before (it has been burned by nothing, ever). It has no mental model of what happens when the code runs. It is producing a sophisticated best guess based on surface patterns.
This means the verification burden doesn’t diminish when you use an AI coding assistant. It shifts. You’re no longer writing the first draft, but your responsibility for catching errors is exactly what it was before, probably higher, because the code arrives fast and fluent and easy to accept. The path of least resistance is to run it, see it pass the obvious test cases, and ship it. That path leads to production bugs that took seconds to generate and hours to diagnose.
The Counterargument
The reasonable pushback here is that modern AI coding environments are increasingly integrated with execution. GitHub Copilot can see your terminal output. Some agentic tools run tests, observe failures, and iterate. Aren’t these systems actually verifying correctness?
Partially. A system that runs tests and observes failures is genuinely better than one that doesn’t. But this creates a new problem: the model is now optimizing for passing the tests you have, not for correctness in general. If your test suite has gaps (and it does, all test suites have gaps), the model can produce code that satisfies every test you run while still being wrong in ways that matter. It’s fitting to your observable signal, not to the underlying requirement. An agent that iterates until tests pass is not the same thing as an agent that understands what the code is supposed to do.
These agentic systems are useful. I’m not dismissing them. But they don’t resolve the fundamental issue; they just add a feedback loop that depends entirely on the quality of your tests.
The Right Mental Model
Treat AI-generated code the way you’d treat code submitted by a contractor who is technically fluent, works at extraordinary speed, and has no domain knowledge of your system. You review everything. You don’t assume correctness because the style is clean. You run it against your actual edge cases, not just the happy path.
The developers I’ve seen get the most value from these tools are the ones who’ve internalized this. They use the AI to do the tedious structural work, the boilerplate, the first-pass implementation of a well-understood pattern, and then they bring genuine scrutiny to the output. The developers who struggle are the ones who use AI fluency as a proxy for correctness.
The tools are real. The productivity gains are real. The verification gap is also real, and pretending it isn’t doesn’t make you a more effective developer. It makes you a faster producer of unreviewed code, which is a different thing entirely.
