The Alerting Paradox
The more alerts a system sends, the less anyone pays attention to them. Past a threshold, adding alerts makes a system less observable, not more — because the alerts that matter drown in the ones that don't.
Operations
The Error Budget
Perfect reliability is the wrong goal. An error budget turns reliability into a number you can spend — and once it's a budget, the argument about whether to ship stops being a matter of opinion.
The Blast Radius
When a system fails, how much else fails with it? The blast radius of a failure is a design property, not an accident. Systems that fail with a small blast radius are easier to recover from, easier to debug, and less expensive to operate.
The Recovery Cost
How long a system takes to recover from a failure is as important as how often it fails. A system that fails rarely but recovers slowly can accumulate more total downtime than one that fails often but recovers fast.
The Runbook Gap
A runbook written the day after an incident captures what you wish you'd known. A runbook written six months later captures what you remember. The gap between those two is where the operational knowledge goes.
The First Month
New systems tend to be most reliable in their first month of operation — not because they're less likely to fail, but because operators are more likely to be watching. Vigilance decays faster than systems do.
The Near Miss
Near-misses are higher-value reliability signals than actual incidents because they surface failure modes without the cost of actual failure. But most teams only run post-mortems on incidents that broke through, so near-miss signals evaporate before anyone learns from them.
Output-First Observability
Most monitoring is built around processes: did it run, did it error, did it use too much memory. Output-first observability flips that — it asks whether the thing that was supposed to be produced exists, is current, and is correct.
The Alert That Arrived Too Late
An alert that fires after a problem has been accumulating for weeks isn't a monitoring system — it's a postmortem trigger. The gap between when the failure started and when the alert fires is where the actual cost lives.
The Recovery Window
When a system has been silent for weeks, recovery isn't just restoration — it's reconstruction. How you handle the gap matters as much as fixing the underlying failure.
The Failure That Stays Quiet
The dangerous failures aren't the ones that throw errors. They're the ones that fail silently, leave no alarm, and only surface as drift you notice later. The defense is building routines that verify state instead of trusting the last run.
The Discipline of the Boring Check
Running the same health checks when everything is fine feels like wasted motion. It isn't. The boring check that almost always passes is what makes the rare failure visible the moment it happens.
One Failure Is Not an Incident
Alert thresholds exist for a reason. A monitoring system that wakes you up for a single transient error isn't protecting you — it's training you to ignore alerts.