Design the Retry, Not Just the Request
The two pieces before this one described retries going wrong from opposite directions — repeating an operation that wasn’t safe to repeat, and repeating it in a way that overwhelms a system already in trouble. Both failures share a pattern worth naming: retry logic was added to an operation that wasn’t designed with retries in mind, as an afterthought wrapped around code that assumed it would only ever run once. The fix in both cases wasn’t a smarter retry library — it was treating retryability as part of the operation’s design from the start, the same discipline this whole thread keeps circling back to: decide the property up front, rather than discovering its absence during an incident.
This reframing changes the order operations get built in. Instead of writing the operation, then separately wrapping it in retry logic as an infrastructure concern, the design questions belong together from the first draft: can this run twice safely, and if not, what identifier makes a second attempt recognizable as a repeat rather than a new event; how should failure be signaled so a caller can tell “definitely didn’t happen, retry freely” apart from “not sure, retry carefully” apart from “definitely happened, don’t retry, here’s the result”; and what should the caller do while waiting, given that its own retry becomes part of somebody else’s incoming load. None of these are retry-library configuration. They’re properties of the operation, decided by whoever designed it, and no amount of clever backoff or jitter at the call site can supply an answer the operation itself never provided.
This is where idempotency keys, informative failure modes, and backoff-with-jitter stop being three separate techniques and turn out to be one coherent answer to one question: what happens when the same logical request is attempted more than once. An idempotency key answers it at the data layer — the second attempt finds the first one’s result instead of redoing the work. An informative failure mode answers it at the protocol layer — the caller learns whether repeating is safe instead of guessing. Backoff and jitter answer it at the traffic layer — repetition, when it does happen, arrives as a manageable trickle instead of a synchronized wave. Each covers ground the others don’t, and an operation that’s genuinely retry-safe usually needs some combination of all three, not because more mechanisms are always better, but because each is answering a different half of “what happens on attempt two.”
The organizations that handle this well tend to make it structural rather than a matter of individual diligence: a shared idempotency-key convention every write endpoint follows, a standard way of classifying errors as retryable or not that’s part of the interface contract rather than left to each caller’s judgment, a house style for backoff that every client library implements the same way so no individual engineer has to rederive jitter math under a deadline. Structural answers matter because retryability is exactly the kind of property that’s easy to skip when it isn’t required, and expensive to add later once callers already have hardcoded assumptions about how retries should be handled — the same asymmetry this entire thread has been describing about assumptions in general, now applied specifically to the question of what happens the second time.
This closes the thread’s arc: a retry is a second request, and a second request deserves the same design attention as the first one got, not less — what it means for this specific operation to be repeated safely, communicated clearly, and absorbed gracefully. Skip that design work and you inherit whatever the retry library’s defaults happen to do, applied uniformly to operations that have wildly different actual risk profiles for being repeated. Do the design work once, at the operation, and every retry from every caller inherits the right behavior automatically — which is the same leverage this thread found in caching, in scale, and in every other place a silent assumption turned out to be load-bearing: fix it at the source, and it stays fixed everywhere the source is used.