AI coding assistants are genuinely useful. If you’ve used GitHub Copilot, Claude, or ChatGPT to write production code, you’ve probably felt the productivity lift. Functions appear in seconds. Boilerplate vanishes. The code looks clean, follows conventions, and often works exactly as described.
That last word is the problem. “As described” is doing a lot of work.
The Confidence Isn’t the Issue. The Gap Is.
There’s a specific failure mode that trips up developers who are new to AI-assisted coding, and it’s not hallucinated syntax or wrong API calls. Those are visible. You run the code and it breaks. The harder problem is when the AI writes code that is technically correct, passes your tests, ships to production, and then fails because of something it had no way to know.
Consider a simple example. You ask an AI to write a function that calculates the difference in days between two dates. It produces a clean, readable implementation. What it doesn’t know is that your application stores timestamps in UTC but displays them in user-local time, that your company’s billing cycle counts the start date as day one rather than day zero, or that a legacy system upstream sometimes sends dates as Unix timestamps instead of ISO strings. The AI gave you exactly what you asked for. The bug lives in the distance between what you asked for and what you actually needed.
This isn’t a criticism of AI coding tools specifically. It’s a structural property of any tool that operates on the description you provide rather than the full system context you carry in your head.
The Knowledge It Doesn’t Have Is the Knowledge That Matters
Large language models are trained on enormous amounts of public code. They’ve absorbed the patterns, idioms, and common solutions that appear across millions of repositories. That’s genuinely valuable. It means they’re excellent at producing idiomatic code in established patterns.
But your codebase is not public. Your business logic is not public. The undocumented assumption your predecessor made in 2019 about how order IDs are generated is not public. The production incident from eight months ago that led your team to add a specific null check is not public. The model has no access to any of this, and it doesn’t know what it’s missing.
This is different from the well-documented problem of LLMs stating false things confidently. That’s about factual accuracy. What we’re talking about here is more subtle: the model generating code that is logically sound given its inputs, but subtly wrong given the full context of your system. As the article Your LLM’s Confidence Has Nothing to Do With Its Accuracy explains, the fluency of the output gives you no signal about whether the underlying assumptions are correct.
The danger increases with code complexity. For a standalone utility function with clear inputs and outputs, the context gap is small. For anything that touches authentication, financial calculations, data migrations, or concurrency, the implicit context is enormous, and the AI has almost none of it.
What Good Review Actually Looks Like Here
The instinct most developers have when reviewing AI-generated code is to check whether it works. Run it, test the happy path, maybe write a unit test. That’s necessary but not sufficient.
A more useful review practice is to explicitly interrogate the assumptions the code is making. Walk through the function and ask: what does this code assume about the input it receives? What does it assume about the state of the system when it runs? What edge cases has it not considered because nothing in the prompt mentioned them?
This is different from normal code review because you’re not just evaluating the logic. You’re evaluating the gap between the problem the AI solved and the problem you actually have. That requires you to hold both models in your head simultaneously, which is work. It’s also the work that adds value.
A practical habit: when you accept a block of AI-generated code, write a comment (even a private one you delete later) that describes one assumption the code makes that the AI couldn’t have verified. Doing this even once per session keeps you in the habit of reading generated code critically rather than as a finished artifact.
The System Prompt Is Not a Full Briefing
Some developers try to solve the context problem by giving the AI more information upfront. More detailed prompts, pasted-in type definitions, excerpts from related files. This helps, but it has hard limits.
You cannot paste your entire codebase into a context window. You cannot anticipate which pieces of tribal knowledge are relevant to the function you’re about to generate. And even when you provide context, you’re still translating your system’s reality into natural language, which introduces its own imprecision. The AI is reasoning about your description of the system, not the system itself.
Tools that connect models directly to codebases (through IDE integrations, retrieval augmentation, or agentic systems that can read files) reduce this gap meaningfully, and that’s a real advance. But they don’t eliminate it. They shift the problem from “the model has no context” to “the model has some context and you still need to know which parts it missed.”
Where to Put Your Trust (and Where Not To)
None of this means you should avoid AI coding tools. The productivity gains are real and the quality of output in well-defined domains is high. What it means is that you need a calibrated view of where these tools are reliable and where they require extra scrutiny.
AI-generated code is most trustworthy when the problem is well-contained and the context is fully specified in your prompt. Parsing a known data format, implementing a standard algorithm, generating boilerplate from a clear schema. These tasks have low context gaps.
AI-generated code needs the most skepticism when it touches business logic that evolved over time, handles edge cases that only appear in production, or integrates with systems that have undocumented behaviors. These tasks have high context gaps, and the code can be fluent and convincing while still being wrong in ways that matter.
The skill you’re building when you use these tools well isn’t prompt engineering. It’s learning to notice the distance between what you described and what you actually meant, before the code ships rather than after. That skill was always valuable. Now it’s essential.
