Consensus Is Expensive
Every distributed system faces the same uncomfortable truth: if you want every node to agree on the current state of the world, you have to pay for it. Not in money, but in latency, availability, and complexity. The more nodes you have, the higher the bill.
The naive mental model of a distributed database is a single database, just spread across machines. Query any node and get the same answer. Update one and the others instantly know. This is strong consistency, and it’s a coherent, useful abstraction. It’s also brutally expensive to maintain at scale.
The alternative, eventual consistency, sounds like something went wrong. The name implies that your system will, at some unspecified future moment, maybe get around to agreeing with itself. That’s an ungenerous reading. What eventual consistency actually means is that the system makes an explicit, deliberate trade: sacrifice the guarantee of instantaneous agreement in exchange for better availability and lower latency. Given enough time without new updates, all nodes converge to the same value. The trick is figuring out when that trade is worth making, and when it isn’t.
What the CAP Theorem Actually Says
Eric Brewer formally described the core tension in 2000, and Gilbert and Lynch proved it in 2002. The CAP theorem states that a distributed system can guarantee at most two of three properties: Consistency (every read gets the most recent write), Availability (every request gets a non-error response), and Partition tolerance (the system continues operating when network messages between nodes are lost).
Here’s the part that often gets glossed over: partition tolerance isn’t optional. Networks partition. They do it regularly. A system that can’t handle network partitions isn’t a distributed system, it’s a single machine pretending to be one. So the real choice, in any production distributed system, is between consistency and availability when a partition occurs.
This is where the nuance lives. CAP describes binary extremes, but real systems exist on a spectrum. The more practically useful framing comes from the PACELC model, proposed by Daniel Abadi in 2012, which extends CAP to also consider the latency-consistency trade-off that exists even when the system is running perfectly and no partition is occurring. Even in normal operation, you’re choosing between faster responses and stronger guarantees.
What “Eventually” Actually Means in Practice
Take Amazon’s shopping cart, which the Amazon Dynamo paper (published in 2007) describes as a concrete design case. The team made a deliberate choice to favor availability over consistency. If two instances of a cart diverge during a network partition, the system allows both writes and reconciles them later, typically by merging the conflicting versions.
The failure mode of a strongly consistent cart is a customer who can’t add items during a brief network hiccup. The failure mode of an eventually consistent cart is a customer who adds an item and briefly doesn’t see it, or in the merge case, ends up with slightly unexpected cart contents. Amazon judged the second failure mode less damaging to the customer relationship than the first. That’s a product decision embedded in a systems decision.
DNS works the same way. When you update a DNS record, the change doesn’t propagate to every resolver on earth simultaneously. It spreads across the network over hours or sometimes days, with different clients seeing different answers during that window. No engineer would call this broken. The trade-off is so obviously correct for the use case that we rarely think of DNS as an eventually consistent system, but it is.
Cassandra, the wide-column database used by companies including Apple and Netflix, makes this trade explicit in its query API. Every read and write can be tuned with a consistency level. At ONE, you’re talking to one replica and getting whatever it has. At QUORUM, you’re waiting for a majority of replicas to respond. At ALL, you need every replica to agree. These aren’t set-and-forget configurations; they’re per-operation decisions, because different operations in the same application can have different consistency requirements.
The Conflicts That Emerge, and How Systems Resolve Them
Eventual consistency’s hard problem isn’t propagation delay. That’s manageable. The hard problem is conflicts: two clients write to the same key at nearly the same time, on nodes that can’t currently talk to each other. When the partition heals, which write wins?
The simplest answer is last-write-wins (LWW): the write with the later timestamp survives. This works fine for many cases and is easy to implement. The catch is that clocks on distributed machines are not synchronized to microsecond precision, even with NTP. Two writes that feel simultaneous can have contradictory timestamps, and one of them silently disappears. For counters, inventory, or anything where both writes represent real user intent, this is genuinely bad.
A more sophisticated approach is vector clocks, which track causality rather than time. A vector clock is a list of counters, one per node, that increments with each write. When comparing two versions of a value, the system can determine whether one version causally precedes the other, or whether they’re truly concurrent and require explicit conflict resolution. Dynamo uses a form of this. Riak built much of its identity around it.
CRDTs (Conflict-free Replicated Data Types) go further by designing data structures where conflicts are mathematically impossible. A grow-only counter, a last-write-wins register, a two-phase set: these are data types whose merge operations are commutative, associative, and idempotent. Order doesn’t matter, duplicates don’t corrupt the result, and every node converges to the same value. Idempotency is doing real structural work here, not just a nice property to have.
Figma uses CRDTs for collaborative editing. Apple’s Notes app syncing across devices is built on similar principles. The appeal is that you can let clients work completely offline and merge later with no manual conflict resolution required, because the data structure’s semantics prevent conflicts from arising in the first place.
When Strong Consistency Is Worth the Cost
None of this means eventual consistency is always the right call. Financial transactions are the canonical counterexample. If your bank balance is an eventually consistent value and two ATM withdrawals happen simultaneously at nodes that can’t currently reach each other, both might succeed. Both deduct from whatever local state each node has. When the partition heals, you have a problem.
For money, inventory that can’t go negative, seat reservations, and any operation where the correctness of a write depends on knowing the current value with certainty, you want strong consistency. Google Spanner achieves external consistency (stronger than linearizability in some definitions) across globally distributed nodes using TrueTime, a combination of atomic clocks and GPS receivers that bounds clock uncertainty to a few milliseconds. It works, and the latency penalty is real.
The cost of Spanner-style strong consistency is why it’s not the default for everything. Schema operations on large databases illustrate the same principle: the stricter the consistency guarantee around a change, the more expensive and disruptive it is to execute.
Reading Your Use Case Correctly
The practical skill in distributed systems design isn’t memorizing consistency models. It’s correctly categorizing operations by what failure mode they can tolerate.
Ask two questions for any operation. First: what happens if two concurrent writes conflict and one is silently lost? If the answer is “a user sees stale data briefly” or “the shopping cart looks slightly off,” eventual consistency is likely fine. If the answer is “money disappears” or “we double-book a resource,” you need stronger guarantees.
Second: what happens if the system refuses to serve this operation because nodes can’t reach each other? If “the page fails to load” or “the feature is unavailable” is catastrophic, availability matters more than consistency for this operation. If serving stale data or refusing the write is acceptable, consistency is worth the availability cost.
Many applications need both, for different operations. A social media platform doesn’t need strong consistency for a like count. It absolutely needs it for financial transactions if it handles payments. Building these differently within the same system is not a contradiction; it’s appropriate engineering.
What This Means
Eventual consistency is not a euphemism for “broken.” It’s a formal model with well-understood semantics, real trade-offs, and decades of production validation. The systems that power DNS, shopping carts, collaborative editors, and mobile sync are all, in various ways, agreeing to disagree temporarily in exchange for staying available and fast.
The engineers who built Dynamo, Cassandra, and Riak weren’t being sloppy about correctness. They were being precise about which kind of correctness their use cases actually required, and building accordingly. Strong consistency where the use case demands it. Eventual consistency where availability and latency matter more than instantaneous agreement.
The hardest part isn’t implementing either model. It’s knowing which one your system needs, and being honest about the failure modes you’re accepting when you choose.
