When something breaks, the most useful question is often: when did it stop working, and what changed between then and now? The answer requires having a reference point — some record of what the system looked like when it was healthy. Without that, you’re not investigating a change from a known good state; you’re trying to reconstruct what good even looked like in the first place, which is a harder problem.

This is the known-good-state problem, and it shows up in two forms. The first is forensic: you’re investigating a failure and you discover there’s no baseline to compare against. Metrics are elevated, but elevated compared to what? Latency is higher, but you don’t know what “normal” latency was before the issue started. The investigation devolves into argument about what the numbers were supposed to be, rather than measurement of how far from baseline they’ve drifted.

The second form is operational: drift goes undetected because there’s no prior state to drift from. A system’s error rate was 0.1% on day one. By month six it’s at 0.8%. No single change caused this; it accumulated gradually. No alert fired, because the alert threshold was set relative to the current error rate, not the original one. The system is objectively worse, but the drift is invisible because there’s no anchor.

The fix for both forms is the same: capture the state of the system when it’s working, and preserve it as a reference point. For metrics, this means storing the baseline at deployment time and checking against that, not just against recent history. For behavior, it means writing down what the expected outputs look like when the system is healthy — not just absence of errors, but the shape of the output, the range of the latency, the distribution of the results. For testing, it means maintaining a snapshot of what the system produced on a known good input, so future changes can be compared against it.

None of this is technically difficult. The hard part is doing it consistently, before the failure, when the system is working and nobody is thinking about what it would look like to investigate a problem six months from now. The known-good-state problem is almost always a problem of foresight rather than capability: the data would have been easy to capture, but nobody thought to capture it until after they needed it.

The deeper issue is that “working” often feels obvious in the moment and fuzzy in retrospect. When the system is running well, the exact shape of “well” doesn’t seem worth documenting. By the time the system is degraded, the exact shape of the prior good state is exactly what’s missing. The interval between when the baseline would have been easy to record and when it would be useful to have is the window where most systems fail to capture it.