Alerting has a paradox at its center: the instinct that makes systems safer in small doses makes them dangerous in large ones. When something can go wrong, the natural response is to add an alert so you’ll know. Do this enough times and you end up with hundreds of alerts, most of which fire on conditions that don’t actually require anyone to do anything. The result isn’t a more observable system — it’s a less observable one, because the alert that signals a real emergency is now indistinguishable from the forty that fire every day and get dismissed without a glance.

The failure mode is alert fatigue, and it’s not a matter of discipline or attention span. It’s a rational response to a signal that has stopped carrying information. If an alert fires and nine times out of ten there’s nothing to do, the correct Bayesian response is to stop treating it as urgent — and people do, whether or not they’d articulate it that way. The problem is that the tenth time is real, and by then the alert has been trained into background noise. The pager that cried wolf doesn’t get quieter; it gets ignored, which is worse, because the ignoring is invisible until the miss.

The fix starts with a hard rule: every alert should require a human to do something. If an alert fires and the correct response is to look at it and go back to sleep, it should not be an alert. It might be a dashboard metric, a logged event, a number that feeds a weekly review — but not a thing that interrupts a person. The test is simple and unforgiving: for each alert, name the action a responder takes when it fires. If you can’t name one, or the action is “acknowledge and ignore,” the alert is noise wearing the costume of signal, and it’s actively degrading every other alert around it.

This means alerting well is mostly about deletion. The tempting direction is always additive — a new failure mode appears, so you add an alert for it. The disciplined direction is subtractive: periodically audit the alerts that exist, find the ones that fire often and get dismissed, and remove them or convert them into something quieter. An alert that has fired two hundred times and prompted action zero times isn’t protecting you; it’s spending your team’s attention and giving nothing back. Removing it is not a reduction in safety. It’s a reduction in noise, and noise is the thing that hides the alert you actually need to see.

The deeper point is that attention is the scarce resource, not information. Collecting more data is nearly free; the cost is entirely in what you ask a human to attend to. A good alerting system is stingy with human attention — it spends it only when spending it changes an outcome. That stinginess feels risky, because every deleted alert is a small bet that its condition won’t matter. But the alternative — a system that alerts on everything — has already made the opposite bet on a much larger scale, and it loses that bet every time a real emergency arrives buried in a pile of noise nobody reads.