A Rollback Is Not a Time Machine
The comforting story about rollbacks is that they’re an undo button: something’s wrong with the new deploy, revert to the old version, the problem goes away. For the code itself that’s often true — the previous binary runs again, the bug stops executing. What the story leaves out is that a rollback only reverts the code. It does nothing to the data the bad code already touched, the messages it already published, the emails it already sent, or the state it already changed in some other system that doesn’t know or care which version of your service called it. The new code stops running the moment you roll back. Its effects do not.
This gap is easy to miss because it’s invisible in the moment that matters most — the rollback itself feels successful, the error rate drops, the dashboard turns green, and the natural conclusion is that the incident is over. Often it isn’t; it’s only half over. If the bad deploy wrote malformed records for twenty minutes before anyone noticed, those records are still malformed after the rollback — the code that would have prevented them is back, but it doesn’t retroactively fix what already got written. If it double-charged a batch of customers, sent a batch of duplicate notifications, or pushed corrupted data into a downstream system, the rollback stops new instances of that damage without touching the instances that already happened. Reverting the cause doesn’t revert the consequences, and treating a green dashboard as proof of full recovery is exactly the kind of premature all-clear that turns a contained incident into a much longer one, discovered piecemeal over the following days as the downstream damage surfaces on its own schedule.
This connects to a thread this blog keeps returning to: code that runs once is easy to reason about, and code whose effects compound or propagate is not, whether that’s a retried write, a cached value, or — here — a deploy that already touched the world before anyone pulled it back. The pattern is the same each time: the visible signal (retry succeeded, cache hit, rollback deployed) is not the same as the actual state of the world, and the gap between them is exactly where the next surprise comes from. A rollback tells you the cause has stopped. It tells you nothing about the consequences that already started.
The practical response is to treat “what did this version already do that a rollback won’t undo” as a standing question for every deploy, not a scramble that starts only after something’s gone wrong. Before rolling back — or as the first action right after, run in parallel rather than treated as the finish line — ask what was written, sent, charged, or propagated while the bad version was live, and whether any of it needs a deliberate second action to actually correct: a backfill for the malformed records, a reversal for the double charge, a follow-up for the systems that received bad data downstream. None of this is a reason to hesitate on rolling back — stopping the bleeding fast is still correct, almost always the right first move. It’s a reason not to declare victory the moment the graph turns green, because the graph is telling you about the cause, and the cause was never the whole incident. The consequences that already escaped are a separate cleanup job, and they don’t go away just because the code that created them did.