There’s a particular kind of frustration in debugging a production incident: by the time you’ve been paged, opened the dashboards, and formed a first guess, the thing you’re chasing has often already stopped. The queue drained, the bad node got recycled, the traffic spike passed. You’re not observing the failure — you’re standing in the room after it happened, looking for what it left behind. Production state is ephemeral in a way that makes live debugging fundamentally different from debugging a program on your desk, where you can set a breakpoint and the world politely freezes while you think.

This changes what “investigation” even means. You’re not going to reproduce the moment; you’re going to reconstruct it from residue. What did the logs capture, what did the metrics sample, what did a trace happen to record before the request that triggered everything completed and vanished. The quality of that reconstruction is decided almost entirely in advance — by what you were recording while things were fine, not by how cleverly you dig once things are broken. The incident is a photograph you didn’t know you were taking, and afterward you get whatever the shutter caught, no more.

The consequence is that the most valuable telemetry is the kind that survives the event. A gauge that shows the current queue depth is nearly useless in the aftermath, because “current” now reads zero and the depth that mattered was twenty minutes ago. What you want is the history: the queue depth over time, the error rate as a series, the state transitions logged as they happened. Point-in-time views answer “how are things right now,” which is the question you can almost always answer some other way. The hard question is “what was happening at 3:47,” and only something that recorded across time can answer it, because 3:47 is gone and it is not coming back.

This is also why reproducing an incident is so prized and so rare. When you can make the failure happen again, on demand, in an environment you control, you’ve converted forensics back into ordinary debugging — you get your breakpoint, your frozen world, your ability to poke and re-run. Most of the effort in a hard incident is really the effort to get there: to find the specific conditions, the exact input, the particular sequence, that makes the ghost reappear. When you can’t, you’re left inferring from traces, building a story that fits the evidence and hoping the evidence was complete enough to rule out the stories that are wrong.

So the throughline of all of this — instrument the questions before you have them, know what normal looks like, keep history rather than snapshots — comes down to a single acceptance: you will almost never be watching at the exact moment it breaks. The system doesn’t wait for you, and the state that explains the failure evaporates on its own schedule. Everything you do beforehand is an attempt to make sure that when the moment passes, as it will, it leaves behind enough of a trace that you can still find out what happened. Good observability isn’t the ability to watch. It’s the ability to reconstruct something you didn’t see.