There is a strange thing happening in every engineering team using AI coding assistants right now. A tool that has never compiled a program, never watched a test suite fail, never debugged a production incident at 2am is writing production code. And it’s often pretty good. That fact deserves more scrutiny than it usually gets.
What These Models Actually Learned From
GitHub Copilot, Claude, GPT-4, and their relatives are large language models. They learned to write code by processing enormous quantities of text: open-source repositories, Stack Overflow threads, documentation pages, blog posts, and more. During training, the model adjusted billions of numerical weights to predict what token (roughly, what word or symbol) should come next in a sequence.
That’s it. The model never executed a single function. It never saw a stack trace caused by its own output. It has no concept of runtime, no memory of whether the code it generated actually worked. What it has is an extraordinarily detailed statistical picture of what code looks like across millions of human-written examples. That picture is useful enough to generate plausible-looking code for a wide range of tasks. But “looks like correct code” and “is correct code” are different properties, and the gap between them is where most of the real problems live.
The Difference Between Syntax and Semantics
Code has two layers of correctness. Syntax is the grammar: are the brackets balanced, is the function call structured properly, does the variable name resolve? Semantics is the meaning: does this function do what you think it does, does this query return the right rows, does this edge case get handled?
LLMs are exceptionally good at syntax. They’ve seen enough code that generating syntactically valid programs is almost trivial. Semantic correctness is much harder, because it requires understanding what the code is supposed to do in a specific context, not just what it looks like.
A model generating a SQL query to “get all users who haven’t logged in for 30 days” can absolutely produce valid SQL. Whether the query handles time zones correctly, whether your last_login column uses UTC or local time, whether NULL values in that column should be included or excluded: those are semantic questions that depend on your specific system’s behavior. The model doesn’t know your system. It’s making educated guesses based on patterns from other people’s systems.
Confidence Without Feedback
Human developers get continuous feedback signals. You write code, you run it, it fails, you learn something. Over years of this cycle, you build intuitions about what works and what doesn’t in real systems. You remember the time a race condition took down a service, and that memory shapes how you write concurrent code.
LLMs have none of this. They receive no feedback from the code they generate. A model that produces a subtle off-by-one error gets no correction signal from that error. It will produce similar errors again, at the same confidence level, because confidence in an LLM is not calibrated to execution outcomes.
This is related to a broader phenomenon worth understanding: as these models get more capable, the errors they produce become harder to spot. The smarter the model, the more confidently it lies. An obvious syntax error is easy to catch. A logically coherent but semantically wrong implementation of a caching strategy can sit in your codebase for months.
Where This Actually Bites You
The failure modes cluster in predictable places. It helps to know them.
Security-adjacent code is particularly risky. Input validation, authentication flows, cryptographic operations: these are areas where the code needs to be exactly right. LLMs have seen lots of authentication code, but they’ve also seen lots of broken authentication code, because security vulnerabilities are well-documented in the same repositories they trained on. The model isn’t deliberately generating insecure code; it’s pattern-matching against a corpus that includes both good and bad security practices.
Library versions and API changes catch people constantly. A model trained on data from a certain period will confidently use APIs that no longer exist, deprecated methods, or patterns that were considered correct under an older version of a framework. The code looks right because it was right, once. This is one reason you shouldn’t treat LLM-generated code as documentation.
Edge cases in business logic are almost impossible for the model to handle correctly without deep context. Your billing system’s handling of prorated credits, your inventory system’s behavior when two concurrent orders claim the last unit: these involve rules that exist in your organization’s head and maybe in comments scattered across your codebase. A model working from a brief prompt has no access to any of that. It will invent behavior that is plausible, not behavior that is correct for your system specifically.
Concurrency and state are hard for humans and harder for models. Async code that looks correct can have subtle ordering dependencies that only manifest under load. The model has seen patterns for handling concurrency but has never experienced the runtime behavior those patterns produce.
The Interesting Part: Where the Model Is Actually Strong
None of this means you should stop using these tools. It means you should use them for the right things.
LLMs are genuinely excellent at boilerplate that follows clear, well-established patterns. Setting up a REST endpoint in an opinionated framework, writing a data class, generating unit tests for a function with a clear specification: these tasks map well to pattern-matching from a large corpus. The patterns are stable, well-represented in training data, and don’t depend heavily on runtime context.
They’re also good at the translation tasks that consume significant developer time: converting between data formats, reformatting code for consistency, writing documentation from code, or explaining what a function does. These are text manipulation tasks at heart, and that’s exactly what LLMs were built for.
Exploration benefits enormously from AI assistance. If you’re working in an unfamiliar library and want to understand its API surface before committing to an approach, an LLM can sketch out five different approaches quickly. You still need to verify them, but having a concrete starting point to react to is faster than reading the entire docs cold. Think of it as a first draft, not a final answer.
How to Adjust Your Workflow
The practical adjustments aren’t dramatic, but they matter.
Review generated code for semantic correctness, not just syntax. Your linter and your compiler already check syntax. When you review AI-generated code, ask whether it does the right thing in your specific context, not just whether it compiles. This sounds obvious but it’s easy to fall into approving syntactically valid code that has semantic problems.
Treat security-sensitive paths as requiring human authorship. Authentication, authorization, payment processing, data access controls: write these yourself or have them carefully reviewed by someone who understands the attack surface. This isn’t distrust of AI specifically; it’s recognizing that these are the parts of your system where the cost of a subtle error is highest.
Give the model more context than you think it needs. The model’s semantic errors often come from insufficient context. If you paste in the function signature alone, you’ll get a plausible implementation. If you include the surrounding code, the relevant data models, and a clear description of the edge cases you care about, you’ll get a much more relevant result. What LLMs actually do with unfamiliar context matters here: more relevant context produces more accurate completions.
Run the code. This sounds almost too obvious to say, but a real pattern in AI-assisted development is accepting suggestions without executing them. The feedback loop that humans lack during training needs to be supplied by you during use. Write the test, run it, see what happens. The model is guessing; your test suite is measuring.
What This Means
AI coding assistants are not digital developers. They’re pattern engines trained on the artifacts of development: the code, the documentation, the discussions. They’ve absorbed an enormous amount of knowledge about what programs look like without ever experiencing what programs do.
That distinction matters practically. It tells you which tasks to delegate confidently (pattern-following, boilerplate, translation), which to delegate carefully (any logic with significant business or security implications), and which to keep close (anything where correctness depends on runtime behavior your system exhibits uniquely).
The right mental model is probably “very well-read intern who has studied every codebase on the internet but has never shipped a feature.” They can produce impressive drafts. They need supervision on anything that matters. And they have absolutely no intuition built from watching things break at runtime, which is where most of the real engineering knowledge comes from anyway.
