There is a particular kind of technical debt that doesn’t show up in your backlog. It doesn’t trigger a failing test or throw an exception. It lives in the gap between what your codebase does and what your engineers actually understand about why it does it.
This is the debt that AI copilots are quietly accumulating for some teams right now.
The Setup
In late 2023, a mid-sized fintech company (one whose story has been shared in engineering retrospectives and anonymized in industry discussions, so I’ll keep it that way) decided to go all-in on AI-assisted development. The pressure was familiar: smaller team than the roadmap demanded, investors watching velocity metrics, a competitor shipping features faster. GitHub Copilot was already in use, but the team moved to a more aggressive posture. Engineers were encouraged to use AI assistance throughout the development cycle, from scaffolding new services to writing tests to handling unfamiliar parts of the stack.
Shipping velocity increased noticeably. Code reviews moved faster because suggestions were syntactically clean and the AI-generated tests gave reviewers confidence. Sprint velocity numbers looked good in standup.
The problems surfaced slowly, then all at once. A senior engineer left the company. She had been the person who pushed back on AI suggestions, who asked “but why is it doing it this way,” who read the generated code carefully and flagged the subtle logic errors. After she left, the team started hitting bugs that took days to diagnose instead of hours. A particularly nasty issue in their payment reconciliation logic took three engineers nearly two weeks to trace. The bug wasn’t new. It had been in a pull request that everyone had approved months earlier, generated mostly by the copilot, reviewed quickly because it looked fine.
When a new engineer joined and asked the team to walk her through how the reconciliation service worked, nobody could do it cleanly. They could show her the code. They couldn’t explain the reasoning behind it.
What Actually Happened
This isn’t a story about AI writing bad code. The code was mostly fine. It’s a story about a team that outsourced comprehension.
There’s a cognitive process that happens when you write code yourself, even when you write it badly the first time. You make decisions. You remember why you reached for a particular approach. You build a mental model of the system as you construct it. When you read someone else’s code, you do a compressed version of the same thing, reconstructing their intent from what they wrote.
When you accept AI-generated code that works, you often skip both steps. You get the output without the reasoning. This is fine occasionally. It becomes a problem when it’s the default.
The fintech team had, without meaning to, created a system where the AI held the reasoning and the engineers held the acceptance criteria. The AI knew (in a functional sense) why the reconciliation logic was structured the way it was. The engineers knew it passed the tests.
This is related to a problem the software industry has known about for a long time. Studies on automation and skill maintenance, including research on pilots and autopilot systems, have shown that humans who rely heavily on automation for routine tasks lose proficiency at those tasks faster than expected. The skill atrophies not because the person becomes incapable, but because they stop practicing the low-level reasoning that builds and maintains competence. Aviation researchers call this “automation complacency.” The phenomenon is documented, real, and not controversial in human factors literature.
Software engineering is different from flying a plane, but the underlying dynamic isn’t that different. If you stop reading code carefully because the AI makes it look trustworthy, you stop building the pattern-recognition that makes you a good code reviewer. If you stop debugging by reasoning through execution paths because the AI usually suggests the right fix, you lose the ability to debug things the AI can’t see. And as we’ve written about before, the bugs AI writes confidently are often exactly the ones that look fine on a surface read.
Why It Matters Beyond This Team
The fintech story is one case, but the pattern shows up in engineering communities with enough regularity that it’s worth taking seriously rather than dismissing as anecdotal.
What makes this pattern particularly tricky is that it’s invisible during the period when things are going well. Velocity is up. Bugs are down, or at least the simple bugs are down. Engineers feel productive. The problems only become visible when you need the understanding that wasn’t built: during a complex incident, during onboarding, when a key person leaves, when you need to make a significant architectural change and nobody can confidently predict what it will break.
The teams that seem most resistant to this pattern are the ones that treat AI output as a first draft rather than a finished product. Not as a draft that needs stylistic polish, but as a draft that needs to be understood. Engineers on these teams read AI-generated code the way they’d read a junior developer’s PR: looking for correctness, yes, but also asking themselves if they could explain it to someone else. If the answer is no, they either figure it out or rewrite it themselves.
This adds friction. It slows down the acceptance step. It partially trades back some of the velocity gain. Teams that do it anyway seem to make a deliberate bet that the comprehension is worth more than the speed, especially in complex or critical domains.
What We Can Learn
The lesson isn’t “don’t use AI copilots.” The lesson is that the metric you’re optimizing for matters enormously, and “lines of code shipped” is a terrible proxy for “engineering capacity built.”
A few concrete things teams seem to get right:
Treat AI suggestions as requiring explanation, not just review. Before merging AI-generated code, someone should be able to explain the non-obvious choices in it. Not defend them, explain them. If nobody can, that’s a signal.
Keep some work intentionally manual. Not as a ritual, but because certain kinds of problems, especially in your most critical systems, are where engineers build the pattern recognition that makes them valuable. Ceding all of that work to an AI is ceding the learning.
Value the people who ask why. The senior engineer who left in the fintech story was doing something that looked like friction: slowing down reviews, pushing back on AI suggestions, demanding explanations. That friction was load-bearing. When it left, the system got brittle.
Recognize that onboarding reveals what you actually know. If a new engineer joins and the team struggles to explain how core systems work, that’s a diagnostic. The knowledge should live in people, not just in the code the AI helped write.
AI assistance is a genuine productivity multiplier for a lot of engineering work. The teams that use it well tend to be the ones where the engineers are still doing the thinking, using the AI to reduce the typing. The teams that use it poorly have quietly inverted that relationship, and they often won’t find out until a moment when they need the thinking most.
