Alert rules almost never get written in advance. They get written after: something broke, someone noticed late, and the retrospective produced an action item to add a check so that next time it’s caught sooner. Repeat that for a few years and you have a monitoring setup that is, quite precisely, a record of every way the system has already failed. That’s not nothing — those failures do recur, and catching them faster is real value. But it means coverage was never designed against the space of possible failures. It was assembled, one incident at a time, from the subset that happened to occur while someone was paying attention.

The gap that leaves is easy to describe and hard to see: the outage that gets you is the one nobody had a threshold for, because nobody had experienced it yet. A dependency starts returning success responses with empty payloads. A scheduled job stops being scheduled at all, so it emits no errors — it emits nothing, and nothing looks identical to quiet. A config change silently disables a code path, and every metric on the dashboard stays flat and green while the feature does nothing. None of these trip a rule, because every rule in the system was derived from a specific past failure with a specific past shape. The absence of alerts is being read as evidence of health when it’s really evidence that this particular failure isn’t one of the ones you’ve catalogued.

The partial fix is to alert on the thing you actually care about rather than on the mechanisms that can break it. There are effectively unlimited ways for orders to stop being processed and exactly one fact that matters: orders stopped being processed. A rule that watches the outcome — successful checkouts per minute, jobs completed per hour, records written since the last run — catches whole categories of causes at once, including the ones nobody has thought of yet, because it doesn’t need to know why the number went to zero to notice that it did. That’s the difference between coverage that grows linearly with your incident history and coverage that holds for failures you haven’t had.

It also means treating silence as a condition to be checked rather than a default to be trusted. A job that runs hourly should be alerting when an hour passes with no completion recorded, not just when it completes with an error, because “produced no output” and “produced a failure” look nothing alike from the monitoring side and only one of them is naturally noisy. The same logic applies to any heartbeat, feed, or pipeline: the healthy state should be something the system actively asserts on a schedule, so its absence is detectable. Otherwise a component can fail by simply going quiet, which is the single most common way to fail without anyone noticing.

None of this makes coverage complete — nothing does, and a system that can only fail in ways you’ve anticipated isn’t a real system. But it changes what the alert set is built from. Instead of a list of remembered incidents, it becomes a small set of claims about what “working” means, expressed in terms a business cares about, with silence treated as a signal rather than as consent. Failures you haven’t seen yet will still find the gaps. The point is to make sure they have to do more work to hide than simply being unprecedented.