The code assistant in your editor is very good at writing code. It is not good at the thing that would tell it whether that code works. Those are different skills, and conflating them is the source of most of the grief developers experience when they start trusting AI output too readily.
Here is the core fact, worth sitting with: a large language model has never executed a single line of code. It has read an enormous amount of code, enough to develop a sophisticated statistical model of what code tends to look like, but it has no runtime, no compiler in the loop, no feedback signal from an actual machine. It predicts tokens. It predicts the next likely character given everything before it. That process can produce code that runs correctly, but when it does, correctness is a byproduct of pattern matching, not a result of verification.
1. The Model Learned From Code, Not From Running Code
This is the foundational distinction. Training a language model on code means showing it billions of examples of source files, Stack Overflow answers, documentation, GitHub commits, and pull request diffs. The model learns statistical relationships between tokens. It learns that after def connect(self): there often comes something involving socket or host or timeout. It learns that SQL queries tend to follow a particular shape. It learns that error handling usually looks a certain way in Python but a different way in Go.
What it does not learn is what happens when you actually execute that code against a production database with 40 million rows, or what a segfault looks like at runtime, or why a race condition only manifests under load. Those are runtime phenomena. The model has representations of text that describes those things, but it has no experience of them.
The analogy that captures this: the model has read every book ever written about swimming. It has a sophisticated understanding of swimming vocabulary, swimming technique descriptions, swimming competition rules. It has never been wet.
2. Confidence Is a Syntax Property, Not a Correctness Signal
One of the stranger properties of LLM-generated code is that it looks equally confident whether it is right or wrong. The formatting is clean, the variable names are sensible, the comments are plausible. There is no visual signal that the model is operating outside its training distribution or hallucinating an API method that does not exist.
This is not a quirk. It is structural. The model generates tokens based on likelihood, and likely tokens are often things that look grammatically and stylistically correct. A plausible-looking but wrong function name gets generated with the same fluency as a correct one. A made-up parameter name is indistinguishable in the output from a real one.
Developers who have not internalized this burn time debugging code that the AI wrote confidently and incorrectly. The confidence of the output has nothing to do with its accuracy, and this is not a bug that future versions will simply fix. It is a property of how the generation process works.
3. It Cannot Reason About Your Specific Runtime
LLMs trained on public code have never seen your codebase. They do not know your dependency versions, your environment variables, your database schema, or the specific behavior of the third-party library your team vendored in 2019 and has not updated since. When the model generates code that integrates with your systems, it is making educated guesses based on what those systems tend to look like in the wild.
This is usually fine for greenfield code using popular, stable APIs. It becomes a real problem for code that needs to interact with your internal abstractions, or with library versions that differ from what dominated the training data. The model will generate code that looks exactly right and targets a slightly different version of the library than you are running. The error you get will be confusing until you figure out what happened.
Context windows help here. Feeding the model your actual schema, your actual function signatures, your actual error messages pulls its predictions toward your reality. But you are doing the work of bridging that gap, not the model.
4. The Bug It Cannot See Is the One It Wrote
Ask an LLM to review code for bugs, and it will often find them. Ask it to review its own recently generated code, and it will frequently miss problems that a different prompt, or a compiler, would catch immediately. This is not inconsistency for the sake of it. The model’s review of code is again a pattern-matching operation. If the code it generated looked like correct code when it was generating it, the same patterns tend to make it look like correct code when reviewing it.
The class of bugs that are particularly hard for models to catch are semantic bugs: code that is syntactically valid, that uses real APIs correctly, but that does the wrong thing. Off-by-one errors, incorrect comparison operators, subtle state mutation in a context where immutability was expected. These bugs AI writes confidently and cannot see are not random noise. They cluster around the gap between syntactic correctness and behavioral correctness.
This is not an argument against using AI for code review. It is an argument for understanding what that review is actually doing, so you know where to apply your own scrutiny.
5. Execution Feedback Is the Thing You Have That It Does Not
Here is where the relationship between developer and model gets more productive if you think about it correctly. You can run the code. The model cannot. That asymmetry means your job is not to evaluate whether the model’s code looks right, but to get it into a runnable state quickly and let the compiler, the test suite, and the runtime tell you what is actually true.
The developers who get the most out of AI coding tools tend to operate in tight loops: generate, run, paste the error back, regenerate. They treat the model as a fast first draft that requires external validation, not as an oracle. The feedback signal from actual execution is information the model genuinely does not have access to and is genuinely useful to it when you include it in your next prompt.
This also means investing in the infrastructure that makes that loop fast. Good test coverage, fast test runners, tight local development environments. The faster you can get from generated code to execution result, the faster the model’s pattern-matching capability can be corrected by reality. That is, fundamentally, a better use of the tool than staring at generated code trying to reason about whether it might work.
The model is doing something genuinely impressive. Predicting plausible, often correct code from a text description is a hard problem that was not solved until recently. But impressive is not the same as reliable, and reliable is what production software needs. The gap between those two things is exactly where your judgment, your tests, and your runtime live.
