The Deprecation That Never Ends
Marking something deprecated is easy. Removing it is the hard part, and most deprecations never get there — they just accumulate, and the old thing runs forever alongside the new one.
Blog
The At-Least-Once Default
Most messaging systems promise to deliver each message at least once, not exactly once. The gap between what you assumed and what the system actually guarantees is where the duplicate-processing bugs live.
The Idempotency Requirement
The moment you add retries to a system, you've made a promise you might not be keeping: that doing the same operation twice is the same as doing it once. Idempotency is what makes that promise true.
The Ordering Assumption
Messages arrive in the order they were sent — until they don't. Assuming global ordering in a distributed system is one of those beliefs that holds in testing and breaks in production, quietly, in ways that are hard to trace.
The Alerting Paradox
The more alerts a system sends, the less anyone pays attention to them. Past a threshold, adding alerts makes a system less observable, not more — because the alerts that matter drown in the ones that don't.
The Error Budget
Perfect reliability is the wrong goal. An error budget turns reliability into a number you can spend — and once it's a budget, the argument about whether to ship stops being a matter of opinion.
The Graceful Degradation Default
When a dependency fails, a system has two options: fail with it, or degrade around it. Most systems fail with it — not because degrading is impossible, but because nobody decided in advance what the degraded state should be.
The Blast Radius
When a system fails, how much else fails with it? The blast radius of a failure is a design property, not an accident. Systems that fail with a small blast radius are easier to recover from, easier to debug, and less expensive to operate.
The Recovery Cost
How long a system takes to recover from a failure is as important as how often it fails. A system that fails rarely but recovers slowly can accumulate more total downtime than one that fails often but recovers fast.
The Runbook Gap
A runbook written the day after an incident captures what you wish you'd known. A runbook written six months later captures what you remember. The gap between those two is where the operational knowledge goes.