Retrying a failed request is one of the most reflexive pieces of resilience engineering — the call didn’t come back, so try again, obviously. What that reflex quietly assumes is that “didn’t come back” means “didn’t happen.” Often it doesn’t mean that at all. A timeout can occur after the operation completed successfully and only the response got lost; a connection can drop after the write landed but before the acknowledgment arrived. From the caller’s side these look identical to a clean failure — no response is no response — but the two situations call for opposite responses. If nothing happened, retrying is exactly right. If something happened and you just didn’t hear about it, retrying means asking for it to happen again, and depending on what “it” is, that’s either harmless or a second charge on someone’s card.

This is why the honest question before adding a retry isn’t “how many times should I retry” but “what happens if this operation runs twice.” The answer sorts operations into two very different categories. Some are naturally safe to repeat — setting a value to a fixed state, reading data, deleting something that’s already gone — because running them once or five times leaves the world in the same place. Others are not — incrementing a counter, sending an email, charging a payment, appending to a log — because each execution is a distinct event that stacks on the ones before it. Retrying the first category is free. Retrying the second category, without more thought, quietly turns a resilience feature into a bug generator, and the bug is specifically the kind that shows up as a customer complaint about a duplicate charge rather than as an exception in a log.

The fix has a name — idempotency — and the idea is simple even though the plumbing takes some care: give each logical operation a unique identifier the caller generates once, and have the receiving side remember which identifiers it has already processed, returning the original result instead of doing the work again when the same identifier shows up twice. This turns “did this run” into a question the system can actually answer, rather than one it has to guess at from the caller’s side based on whether a response happened to arrive. With that in place, retrying becomes safe again, because a retry with the same identifier isn’t a second event, it’s a second inquiry about the same event — which is exactly the distinction that was missing when the system had no way to tell “new operation” from “unanswered old one” apart.

Retries interact with the caching thread in an instructive way, too: a naive retry-with-backoff strategy is itself making an implicit promise, the same way a cache does — that repeating the call is equivalent to the call having happened once. That promise is only true for operations that are actually idempotent, and just like a cache’s invalidation strategy, the promise is invisible in the code that benefits from it. A retry wrapper looks the same whether it’s wrapping a safe read or an unsafe charge; nothing about the retry logic itself tells you which one you’re looking at, so the safety has to come from a decision made about the specific operation, not from a general policy applied uniformly to everything that might fail.

None of this argues against retrying — transient failures are common and retries are frequently the right response to them. It argues for asking the “runs twice” question before reaching for the retry, the same way this thread has repeatedly argued for surfacing an assumption before building on top of it. An operation that’s safe to repeat deserves a generous retry policy with real confidence. An operation that isn’t needs either an idempotency key before it gets one, or a much more conservative hand — because the failure mode of over-retrying an unsafe operation isn’t “slightly wasteful,” it’s “did the thing that was never supposed to happen twice, happen twice,” and that’s a much harder story to tell someone after the fact.