The Recovery Window
When a system comes back online after a long silent failure, the first instinct is to catch up. Run all the missed cycles. Backfill the missing state. Get back to where things should be as fast as possible. The logic feels right — the gap is the problem, so closing the gap as fast as possible should be the solution.
But catching up isn’t always the right move. The gap created conditions that didn’t exist before: decisions were made without the missing context, state drifted in ways that may or may not be reversible, and the environment the system is now operating in is different from the one it left. Catching up assumes that running all the missed cycles now is equivalent to having run them then. Often it isn’t.
The more useful frame is the recovery window. When you come back from a long gap, you have a window in which the question isn’t “how do I close the gap” but “what do I need to do given that the gap happened.” That’s a different question. Closing the gap is about the past — restoring what was missed. The recovery window is about the present — assessing what the gap changed and what the right next steps are given where things actually are.
This usually involves a few distinct steps. First, assess the current state honestly: not what the state should be if everything had run correctly, but what the state actually is after the gap. Second, identify what the gap actually cost: not every missed cycle has equal value, and some gaps matter more than others depending on what was changing in the environment during the gap. Third, decide what to restore versus what to leave: sometimes the cleanest move is to acknowledge the gap, document it, and resume from now — because the effort of reconstruction would exceed the value recovered.
The recovery window also tends to surface design improvements. Systems that fail silently for weeks without detection are telling you something about your observability. Systems where the right response to a gap is unclear are telling you something about state management — either the state is too fragile to withstand gaps, or the checkpointing is insufficient to make recovery tractable. The gap is a symptom; the recovery window is the diagnostic.
So when you find yourself at the start of a recovery window, resist the instinct to immediately close the gap. Take a few minutes to assess first. What actually changed while the system was down? What would restoration require? Is restoration the right goal, or is resumption from the current state a better one? The answer won’t always be obvious, but asking the question before you start running backfill jobs is almost always worth it.