Retries are the most common reliability mechanism there is. A request fails, so you try it again; a call times out, so you send it once more. It’s such an obvious response to transient failure that it often gets added without much thought — a retry loop here, a client library with automatic retries there. But retries carry a hidden requirement, and systems that add them without meeting it don’t become more reliable. They become more likely to do the wrong thing twice.

The requirement is idempotency: doing an operation more than once has the same effect as doing it once. When an operation is idempotent, a retry is free — worst case, you repeat work that was already done and nothing changes. When an operation is not idempotent, a retry is dangerous, because the first attempt might have succeeded even though the response never arrived. A payment that times out might have gone through; retrying it charges the customer twice. An email send that returns an error might have already delivered; retrying it sends a duplicate. The failure that triggered the retry and the success that actually happened are indistinguishable from the caller’s side, and that ambiguity is exactly where the damage lives.

This is why the interesting question about any operation is not “does it work?” but “what happens if it runs twice?” For reads, the answer is almost always “nothing bad” — reading the same data twice is harmless. For writes, the answer depends entirely on how the write is structured. Setting a field to a specific value is idempotent: set it twice, it’s still that value. Incrementing a counter is not: increment twice, you’ve counted something once as twice. The difference isn’t in the reliability machinery around the operation; it’s in the shape of the operation itself, and it has to be designed in.

The standard mechanism for making non-idempotent operations safe under retry is an idempotency key: a unique identifier the caller attaches to each logical operation, which the server records the first time it processes and recognizes on any repeat. The second time a request arrives with a key the server has already seen, it returns the original result instead of doing the work again. This turns a naturally unsafe operation — charge this card, create this order — into one that can be retried freely, because the server, not the network, becomes the source of truth about whether the operation already happened. The key is what lets the server tell “this is a retry” apart from “this is a new request that happens to look identical.”

The broader lesson is that reliability mechanisms have preconditions, and adding the mechanism without meeting the precondition is worse than not adding it at all. Retries assume idempotency. Caching assumes the cached value is still valid. Failover assumes the backup is consistent with the primary. Each of these is a genuine improvement when its assumption holds and a new class of bug when it doesn’t. The discipline is to notice the assumption before turning on the mechanism — to ask, before adding the retry, whether the thing being retried can survive being done twice, and to make it so if it can’t.