The software industry has been studying its own scheduling failures for decades. The Standish Group has tracked project outcomes since the 1990s. Researchers like Daniel Kahneman have named the underlying psychology. And yet, across companies of every size and sophistication, projects routinely land at two to three times their original estimates. This pattern is too consistent to blame on incompetence. The real explanation is simpler and more uncomfortable: most project estimates are not honest forecasts. They are social agreements.
Estimates Are Negotiations, Not Predictions
When a product manager asks an engineering team how long something will take, both parties understand the stakes. A number that sounds too high kills the project or the team’s reputation for efficiency. A number that sounds right gets the work approved and preserves relationships. The estimate that gets written down is almost never the engineer’s true internal forecast. It’s the number that allows the meeting to end well.
This isn’t cynicism. It’s how organizations actually function under deadline pressure and resource scarcity. Kahneman’s research on the planning fallacy showed that people systematically underestimate task duration even when they have direct experience with similar failures. But in software, the planning fallacy is compounded by an additional force: nobody wants to be the person who killed the roadmap by saying it would take eighteen months.
The result is what you might call the optimism tax. Every project pays it, and nobody budgets for it.
The Problem Gets Buried in the Definition of “Done”
Software estimates routinely exclude the work that turns out to be most of the work. Initial scoping captures the happy path: write the feature, ship it, move on. It rarely accounts for edge cases discovered during testing, compliance reviews, performance degradation under real load, integration problems with adjacent systems, or the three rounds of stakeholder feedback that reframe the core requirement entirely.
These aren’t surprises. They are predictable categories of work that experienced engineers know will appear. The reason they don’t appear in the estimate is partly cognitive (they’re hard to visualize in advance) and partly political (listing them makes the estimate look padded). So they get deferred to the phase nobody scheduled.
This connects to something worth naming directly: the code that gets deleted is often more valuable than the code that gets written. Refactoring, rethinking, and removing wrong decisions from earlier sprints is real work that consumes real time. Estimates almost never include it.
Scope Creep Is a Symptom, Not the Disease
The conventional diagnosis for timeline overruns is scope creep: requirements changed, stakeholders added features, the project expanded beyond its original boundaries. This framing puts the blame on process failure and sells the solution as better requirements documents and more disciplined change management.
That diagnosis is wrong, or at least incomplete. Scope creep happens because the original scope was a fiction. When a team doesn’t deeply understand the problem they’re solving at the start, the real requirements emerge during construction. The feature that seemed clear in the kickoff meeting reveals its complexity when someone actually starts building it. The “simple” integration turns out to require a custom parser. The UX that looked finished in mockups confuses every user in testing.
This is not a failure of documentation. It’s the nature of building software for problems that haven’t been fully understood yet. The solution isn’t more rigorous upfront planning. It’s honest acknowledgment that the estimate at the start is a rough order of magnitude, not a contract.
Organizations Punish Accuracy and Reward Optimism
Here is the structural trap: in most organizations, the engineer who estimates accurately gets penalized and the one who estimates optimistically gets rewarded, at least until the project is underway.
Accurate estimates look slow. They make the roadmap look thin. They raise uncomfortable questions about team capacity. Optimistic estimates keep leadership happy, secure budget approval, and generate the kind of momentum that feels productive. The cost arrives later, when the timeline slips, by which point the original estimate has been forgotten and the conversation shifts to what went wrong during execution.
This incentive structure is not an accident. It is the predictable output of organizations that treat timelines as commitments rather than estimates. When a date becomes a promise, the only socially acceptable behavior is to promise an achievable date, which means a date short enough to be approved and long enough to be plausible, not long enough to be accurate.
The Counterargument
The most serious pushback to this argument is that some organizations genuinely have improved their estimation accuracy through better tooling, more granular task breakdown, and iterative development methodologies. Agile, specifically, was designed to make the estimation problem manageable by working in short cycles rather than predicting far-future outcomes.
This is real progress, and it’s worth acknowledging. Sprint-based development does reduce the damage from bad estimates by catching problems earlier. But it doesn’t eliminate the social dynamics that produce bad estimates in the first place. Teams still anchor quarterly roadmaps to optimistic assumptions. Velocity calculations still get gamed when leadership watches them closely. The unit of deception just gets smaller.
Better process helps. It doesn’t solve a human problem with a procedural fix.
What Would Actually Help
The fix is not a new methodology. It’s a shift in what organizations claim to want from their engineers. If leadership genuinely accepts that early estimates are rough and that accuracy improves over time rather than at the start, the incentive to lie shrinks. If post-mortems focus on improving the estimation process rather than assigning blame for the timeline slip, engineers can afford to be honest.
None of this is technically difficult. It requires organizations to stop treating project estimates as commitments and start treating them as data that improves with iteration. The companies that have internalized this are better at shipping. The rest keep holding retrospectives about why the last project took three times as long, and then running the same process for the next one.
