In 2011, a team at a mid-sized financial services company spent six weeks hunting a bug that would have been unremarkable except for one property: it disappeared whenever they tried to observe it. The application, a high-throughput order-processing system, would occasionally drop transactions. Not crash. Not log an error. Just quietly lose work, the way a conversation gets interrupted and both people politely pretend the sentence was finished.
The bug was intermittent, customer-reported, and statistically reproducible only under production load. Which meant it was, in practice, nearly invisible.
The Setup
The system was written in Java and processed financial transactions concurrently across a thread pool. The team had built it well by the standards of the day. They had logging. They had unit tests. They had a staging environment. They had, in short, all the things you are supposed to have and which do not help with this class of bug.
When a transaction dropped, there was no exception in the logs. The thread had simply moved on. The root cause, eventually, was a race condition in the queue handoff between two components. Under normal load, the timing worked out. Under high concurrency, a thread could read a “task available” flag, get context-switched out before acquiring the lock, and return to find the task already claimed and discarded by another thread. The original thread then marked the slot as processed anyway.
This is a classic time-of-check to time-of-use (TOCTOU) error. The problem is not that it is obscure. The problem is that it is nearly impossible to reproduce under controlled conditions, because controlled conditions change the timing.
What Happened
The team’s first instinct was reasonable: add more logging. They instrumented the queue, added timestamps, and re-deployed to staging with a load test running. The bug disappeared. Not intermittently. Completely.
This is the part where engineers lose weeks. The logging had added just enough latency to each operation that the critical timing window closed. The threads no longer collided. The system worked perfectly, and every log showed exactly why: nothing was going wrong.
They removed the logging and the bug returned in production. They added the logging back and it vanished in staging. They spent two weeks in this loop before someone named the phenomenon explicitly in a team meeting: “We’re practicing observer effect on our own software.”
The observer effect in physics refers to the fact that measuring a quantum system necessarily disturbs it. The software equivalent is less mystical but equally frustrating. A debugger pauses execution. A logging call adds latency. A profiler samples at intervals that change thread scheduling. The act of looking changes what is there to see.
Their eventual solution involved two pieces. First, they rewrote the queue handoff using java.util.concurrent.atomic classes to make the check and the claim a single atomic operation, eliminating the window entirely. Second, they built a separate diagnostic mode that could record a ring buffer of state transitions in memory without flushing to disk, so they could capture the system’s behavior without altering its timing. The in-memory recording added nanoseconds rather than microseconds. The window was tens of microseconds wide. It was enough.
The fix took a day to write once they understood the problem. Understanding the problem took six weeks.
Why This Matters
This story keeps repeating because the tools we reach for first (loggers, debuggers, profilers) are specifically the ones that alter timing-sensitive behavior. And concurrency bugs, by their nature, are timing-sensitive.
The deeper issue is that our mental models of software tend to be sequential. We read code top to bottom. We reason about it one operation at a time. Concurrency bugs exist in the spaces between operations, in the moments when one thread has done half a thing and another thread acts on that half-finished state. Why fixing a production bug is harder than writing the code gets at part of this: production environments have properties that staging environments systematically lack, and load-dependent timing is near the top of that list.
There is also a category problem in how teams classify these bugs. An intermittent issue that only appears under load gets treated as a reliability concern, not a correctness concern. It gets lower priority. It gets investigated when people have time. Meanwhile, in production, transactions disappear.
For software that uses AI components, this problem acquires a new dimension. Non-deterministic outputs mean you cannot always distinguish a timing-sensitive bug from natural variance. If your model occasionally returns an unexpected result, is that the model, the prompt, a caching layer, or a race condition in the state you handed the model? The observation tools available for neural network behavior are even more distorting than a Java debugger. As the prompt you write isn’t the prompt the model reads explains, significant transformation happens between your intent and what the system actually processes, and most of that transformation is silent.
What We Can Learn
The practical lessons from the financial services case are specific enough to be actionable.
Design for atomicity at the boundaries. The TOCTOU error exists because check and claim were separate operations. Wherever two components hand off work, the handoff should be a single atomic operation or protected by a lock held across both steps. Java’s java.util.concurrent package, Go’s channels, Rust’s ownership model – these exist precisely to make correct concurrent code expressible in the language rather than reliant on discipline.
Build diagnostic tooling that respects timing. High-frequency, in-memory ring buffers are the right tool for capturing state in timing-sensitive paths. Writing to disk, formatting strings, or acquiring logging locks all introduce latency that changes the system you are trying to observe. If your only diagnostic tool adds milliseconds of latency to a microsecond-scale race, it will never show you the race.
Take intermittent production bugs seriously earlier. The team’s six-week investigation was extended by treating the bug as lower-priority because it was intermittent. In hindsight, “only happens under production load” is a strong signal that the bug lives in a timing-sensitive path, which is exactly where the most dangerous concurrency errors live. An intermittent data-loss bug in a financial system is not a minor issue because it is intermittent.
Distinguish absence of evidence from evidence of absence. The bug disappearing in staging was not evidence it was fixed. It was evidence that staging had different timing properties than production. These are not the same finding, and conflating them is how teams lose weeks.
The underlying principle is that software is not a static artifact you read and reason about. Under concurrency, it is a dynamic system whose behavior emerges from timing, load, and the interaction of components that were probably designed and tested in isolation. Bugs that live in that emergent behavior are genuinely hard to see, not because engineers are careless, but because the observation tools we have are built for a simpler model of what software is.
The bug that disappears when you look at it is not misbehaving. It is showing you exactly how it works.
