The Bug That Isn’t a Bug
In 2011, a well-documented outage at a major payment processor caused thousands of customers to be charged multiple times for a single transaction. The root cause wasn’t a rogue algorithm or a corrupted database. It was a retry loop: a server timed out, the client assumed the request had failed and sent it again, and the system processed both. The underlying code was, by every conventional measure, correct. The problem was architectural.
This failure mode has a name, and it’s older than cloud computing. It’s a violation of idempotency, the property that says performing an operation multiple times produces the same result as performing it once. The concept comes from mathematics (an idempotent function is one where f(f(x)) = f(x)) but its importance in distributed systems is entirely practical. Networks drop packets. Servers restart mid-request. Load balancers time out and retry. In any system with more than one node, the question is never if a message will be delivered more than once. It’s when.
Most engineers know the word. Few design for it from the start.
Why Distributed Systems Make This Inescapable
A single-process program running on one machine has a clean execution model: an operation either completes or it doesn’t, and the program can usually tell which. Distributed systems break this assumption at the foundation.
The core problem is what researchers call the “dual failure” scenario. When a client sends a request to a server and receives no response, it cannot determine whether the request never arrived, arrived and failed, or arrived, succeeded, and the response was lost in transit. From the client’s perspective, all three cases look identical: silence. The only safe response, in a system that needs reliability, is to retry. But retrying a non-idempotent operation on a request that already succeeded is how you charge a customer twice.
This is compounded by the scale at which modern systems operate. Stripe processes millions of API calls daily. AWS Lambda executes functions that may be triggered multiple times by the same event due to its “at-least-once” delivery guarantee, which is documented explicitly in their developer guidelines. Apache Kafka, one of the most widely used message streaming platforms, defaults to at-least-once delivery semantics for the same architectural reason: guaranteeing exactly-once delivery is expensive and often impossible without cooperation from the consumer. The platforms that run much of the internet are, by default, built on the assumption that your code can handle seeing the same message more than once.
If it can’t, that’s your problem to fix.
The Two Failure Modes Engineers Actually Ship
There are two common ways that non-idempotent code escapes into production, and they look very different.
The first is naive state mutation. Consider a simple endpoint that increments a counter: UPDATE accounts SET balance = balance - 100 WHERE id = ?. A single retry doubles the deduction. The fix isn’t complicated (compare-and-set operations, conditional updates, or storing the intended final value rather than a delta), but it requires thinking about the problem in advance. Developers who design for the happy path first rarely circle back.
The second failure mode is more subtle: operations that are individually safe but become unsafe when composed. Sending a confirmation email is idempotent in isolation if you check for duplicates before sending. Charging a card is idempotent if you track whether the charge was processed. But a workflow that chains these operations can fail halfway through, leaving the system in a state where the email was sent but the charge wasn’t, or the charge succeeded but no record was written. Retrying the whole workflow sends a second email and attempts a second charge.
This is why idempotency can’t be bolted on. It has to be a design constraint from the beginning, applied not just to individual operations but to the transactions that compose them.
Idempotency Keys: The Industry’s Standard Answer
Stripe popularized what is now the dominant practical solution: the idempotency key. When a client initiates a request, it generates a unique identifier (typically a UUID) and includes it in the request header. The server stores the key alongside the result of the operation. If the same key arrives again, the server returns the stored result without re-executing anything.
This approach has two significant properties. First, it moves the responsibility for uniqueness to the client, which is the only party that knows it’s retrying. Second, it decouples the operation’s outcome from the number of times the request is made. The server doesn’t need to know whether this is the first or fifth attempt. It just checks the key.
The implementation details matter. Idempotency keys need to be stored durably, not just in memory, or a server restart defeats the entire mechanism. They need expiration policies, because storing every key forever is impractical. And they need to be scoped correctly: a key that’s unique per user but not per operation type is useless.
Many payment and messaging APIs now require idempotency keys for state-changing operations. Stripe’s API documentation makes this explicit, noting that keys are safe to retry for up to 24 hours. Twilio, PayPal, and many others have adopted similar patterns. The key insight is that this isn’t clever engineering. It’s the minimum viable contract for a reliable distributed system.
Where HTTP Gets It Partly Right
HTTP was designed with a form of idempotency in mind, though it’s often misunderstood. The HTTP specification defines GET, PUT, DELETE, and HEAD as idempotent methods, meaning that making the same request multiple times should have the same effect as making it once. POST is explicitly non-idempotent.
This is a reasonable design, but it only describes intent, not enforcement. A developer can write a PUT endpoint that appends to a log instead of replacing a record, violating the contract. An HTTP proxy that retries a DELETE request because it timed out will delete the resource on the second attempt only to receive a 404, which is actually the correct behavior for an idempotent operation. The semantics are specified; the implementation is not.
REST APIs add another layer of confusion. The community norm is that POST creates resources and PUT replaces them, but many real-world APIs use POST for operations that should be idempotent (like “send this notification”) and PUT for operations that aren’t (like “apply this patch”). The HTTP method becomes a loose convention rather than a guarantee.
The practical implication is that you cannot rely on HTTP’s idempotency definitions to protect your system. They provide a framework for thinking about operations, not a safeguard against duplicate execution.
Database Transactions Aren’t the Full Answer Either
A common response to these problems is “just use transactions.” It’s not wrong, but it’s incomplete.
Database transactions handle atomicity: an operation either fully completes or fully rolls back, leaving no partial state. This is valuable, and systems that lack it have their own serious problems (as covered in detail on this site’s piece on database deletion). But transactions don’t address the duplicate execution problem. A transaction that completes successfully and whose response is then lost in transit will be retried. The transaction will run again, successfully, and now you have two completed operations.
The combination you need is transactions plus idempotency keys. The transaction ensures that each individual attempt is atomic. The idempotency key ensures that successful attempts are recognized and not re-executed. Without both, you have partial safety.
Distributed transactions (two-phase commit, Sagas) add another layer of complexity that most teams don’t actually need. The more common and more tractable problem is making individual service operations idempotent, which can be done without coordinating multiple databases.
The Business Cost of Getting This Wrong
The technical argument for idempotency is clear enough. The business argument is more direct.
Duplicate charges are the most visible failure, but they’re not the only one. Duplicate notifications erode user trust faster than almost any other bug class because they’re visible and repetitive. A user who receives the same “your order has shipped” email three times doesn’t think “there was a retry storm”; they think the product is broken. Duplicate inventory decrements can oversell products that don’t exist. Duplicate webhook deliveries can trigger actions in third-party systems that can’t be undone.
The repair cost compounds the initial failure. Issuing refunds for duplicate charges requires customer service time, payment processor fees, and occasionally disputes or chargebacks. Rebuilding trust after a notification flood requires communication and often compensation. And the engineering time spent investigating “why did this run twice” is engineering time not spent on anything else.
The teams that build idempotent systems by default don’t just have fewer incidents. They have simpler runbooks, more confident deployments, and retry logic they can actually trust. The engineering investment in getting idempotency right is small relative to the cost of getting it wrong at scale.
What This Means
Idempotency is not an advanced topic. It’s a foundational property that every networked system should be designed around, because the alternative, assuming that operations execute exactly once, is not a conservative assumption. It’s a bet against physics.
The practical checklist is short. Assign idempotency keys to all state-changing operations. Store results durably alongside those keys. Use conditional writes instead of blind mutations. Design workflows so that partial completion can be detected and safely retried from the point of failure. Treat “at-least-once” delivery as the default assumption for any message queue or event stream.
None of this is exotic. What’s surprising is how rarely it gets prioritized until after the first incident. The systems that handle failure gracefully aren’t the ones with the most sophisticated error handling. They’re the ones that designed for the possibility of repetition before they shipped.
