Building distributed systems is one of those disciplines where you spend years learning lessons that, in retrospect, were obvious. The frustrating part is that most teams rediscover the same handful of failure modes independently, at great cost. You don’t have to.
These aren’t exotic edge cases. They’re the recurring patterns that show up in post-mortems, in conference talks, and in the 2am pages that ruin weekends. Once you know what to look for, you’ll start seeing them everywhere.
1. They Assume the Network Is Reliable
Peter Deutsch’s classic list of distributed computing fallacies starts here, and with good reason: this assumption kills more systems than any other. Networks drop packets. Connections time out. Routers go down mid-request. A response that never arrives looks identical to a response that’s just slow, and your system needs to handle both cases correctly.
The practical fix is to design around failure explicitly rather than treating it as an exception path. Use idempotency keys so retried requests don’t duplicate work. Set aggressive timeouts with exponential backoff rather than waiting indefinitely. Build circuit breakers that stop hammering a failing dependency and give it room to recover. None of this is glamorous, but systems that treat network failure as routine are the ones that stay up.
2. They Let One Slow Node Drag Everything Down
In a distributed system, you’re often only as fast as your slowest dependency. This is the tail latency problem, and it’s subtler than it looks. When you fan out a request to ten nodes and wait for all ten to respond, the latency of the overall operation is the maximum latency of any single node, not the average. Add enough nodes and you’ll almost always be waiting on an outlier.
Google’s research on their internal systems showed that fan-out requests to hundreds of machines would routinely be delayed by stragglers, even when individual node latency looked fine in aggregate. Their solution was the “hedged request” pattern: if a response doesn’t arrive within a threshold, send the same request to a second node and take whichever responds first. You trade a small increase in load for a meaningful reduction in tail latency. This isn’t always appropriate, but understanding the tradeoff gives you a real tool to work with.
3. They Build for Happy-Path Ordering
Messages in a distributed system don’t arrive in the order you sent them. Events that you think are sequential can be processed out of order, duplicated, or lost entirely. This matters most when your application logic assumes causality: “the order must be placed before it can be shipped.”
The honest fix is to stop pretending your system has a single global clock and start reasoning explicitly about ordering. Use vector clocks or logical timestamps if you need to reason about causality. Design consumers to be idempotent so processing a message twice doesn’t corrupt state. If strict ordering genuinely matters, acknowledge the cost of enforcing it (usually a significant throughput reduction) and decide deliberately whether it’s worth paying.
4. They Mistake Coordination for Reliability
This one runs counter to intuition. When something feels fragile, the instinct is to add more coordination: consensus protocols, distributed locks, two-phase commits. These tools solve real problems, but they also introduce new failure modes and become bottlenecks under load. A distributed lock that becomes unavailable takes down everything that depends on it.
The better question to ask is whether you actually need strong consistency, or whether you’ve assumed you do. Many operations that feel like they require a lock can be redesigned using optimistic concurrency, conflict-free replicated data types, or careful idempotency. Strong coordination should be a deliberate architectural choice, not a default reflex.
5. They Treat Configuration as Stable State
Service discovery, load balancer configs, feature flags, database connection strings: these change, and when they change at the wrong moment, they can cause cascading failures. A misconfigured deployment that gets pushed to production and then rolled back might leave some nodes on the new config and others on the old one. Now you have a fleet that’s partially split, and debugging it is a nightmare.
Make configuration changes observable. Log what configuration each node loaded at startup and surface it somewhere you can query. Use gradual rollouts for config changes the same way you would for code changes. If your system can’t tell you what configuration it’s currently running, you’re operating blind during the moments when it matters most.
6. They Ignore the Thundering Herd
You bring a service back up after an outage, and immediately it falls over again. Every client that was waiting retries simultaneously. Every cache that was warm is now cold. Every queued request fires at once. This is the thundering herd, and it turns a recoverable failure into an extended one.
The standard mitigations are jitter and backoff. Instead of every client retrying after exactly 30 seconds, they retry after 30 seconds plus a random offset drawn from some distribution. Cache warming on startup, before you start serving traffic, prevents the cold cache problem. Rate limiting at ingress gives you a circuit breaker against your own clients. These are all things you need to build before you’re in an outage, not during one. The teams that come back from failures cleanly are the ones who thought through the re-entry problem during calm conditions.
Distributed systems are genuinely hard, but they’re not mysterious. The failure modes are predictable, the mitigations are known, and most of the work is just being honest about where you’ve cut corners and going back to fill them in. Start with the failure mode that’s most likely to affect your system right now, implement one mitigation, and build from there.
