Only One Side Leaves a Record
Access lasts too long and arrives too wide for the same reason: granting produces an artifact and needing produces nothing, so every correction has to argue from silence.
Observability
Can't Reproduce Is a Measurement
Failing to reproduce a bug tells you something real. It just doesn't tell you the thing most people close the ticket believing it said.
The Only Failure That Counts
Both halves of reproduction — succeeding and failing — quietly replace the reporter's failure with yours. Only one of the two is the bug.
The Repro Is a Model
Getting a bug to reproduce feels like the end of the investigation. It's the point where you quietly substitute your version of the failure for theirs.
One Choice, Two Costs
Picking a metric and starving the work it can't see aren't two separate problems. They're the same act, viewed from either side.
The Metric Stopped Being the Thing
A proxy measurement is only honest for as long as nobody is optimizing against it yet. The moment it becomes the target, it starts drifting away from the thing it was supposed to represent.
The Work That Never Shows Up
Every measurement system creates two categories: work that counts and work that doesn't. The second category is where prevention lives, and it starves quietly.
You Are in a Queue You Cannot See
The person you asked is holding eleven other requests. You can't see them, so you assume you're the only one — and that assumption is what makes the queue grow.
The Copies You Forgot About
Deleting a record is easy. Deleting every derivative of that record is the part nobody scoped.
The View Somebody Saved
Once people can filter and search, they start building. What they build becomes shared infrastructure nobody planned to maintain.
Every Notification Spends Attention
Notifications get added one feature at a time, each individually justified. Nobody owns the total, and the total is what determines whether any of them get read.
Nobody Is Sure This Is Unused
Code accumulates not because anyone wants it, but because removing it requires certainty nobody has. The fix is making that certainty cheap to obtain.
Someone Has to Be Able to Answer This
"Who can see this record, and why?" is the question a permission system exists to answer. If nobody can answer it without reading code, the system has already failed.
Write Down What the Data Means
A column name tells you what something is called. It rarely tells you what counts, what's excluded, or which of three plausible definitions the number actually uses.
Follow One Request All the Way Through
Well-written, well-rationed log lines still fail if you can't assemble them into one story. The capability that makes logs worth keeping is being able to trace a single request end to end.
Logging Everything Is Not a Strategy
"Log it just in case" feels like insurance. What it actually buys is a haystack, a storage bill, and a search that times out during the incident you bought it for.
Logs Are Written for the Wrong Reader
Most log lines are written by someone who already knows what the code does, for a reader who doesn't and won't be able to ask.
Nobody Knows What the Config Actually Is
The value a service runs with is assembled from defaults, files, environment variables, a remote store, and per-tenant overrides. Very few systems can tell you what won.
Every Alert Is a Claim About a Person
An alert isn't a statement about a metric. It's an assertion that a specific human should stop what they're doing and act — and most alerts were never designed to earn that.
The Response Is Part of the Alert
An alert that fires correctly and leaves the person receiving it with no idea what to do has done half a job. The response isn't downstream of the alert — it's the reason the alert exists.
You Alert on the Failures You've Already Had
Alert rules accumulate one incident at a time, which means your coverage is a map of your history — not of the ways your system can actually break.
Write the Error for the Person Reading It
An error message is written once, in a moment of frustration, by someone who already knows what went wrong. It's then read by people who don't — often at their worst moment, with no other information to go on.
Shorten the Loop Before You Optimize It
When work feels slow, the instinct is to get better at the steps inside the loop. Usually the bigger win is making the loop itself shorter — so being wrong stops costing so much.
The Code You Can't Loop On
Some code has no fast feedback loop at all — you can't easily run it, watch it, or reproduce its failures. That code doesn't just move slowly. It resists being understood, and the slowness compounds.
The Loop Is the Unit of Speed
How fast you build isn't set by how fast you type. It's set by how quickly you can go around the loop of making a change, seeing what it did, and learning from it. That cycle time is the real speed.
Normal Is a Measurement
A number from production means nothing until you know what that number usually is. The hardest part of observability isn't collecting metrics — it's knowing what normal looks like, because 'bad' is defined entirely by contrast with a baseline you had to measure first.
The Incident Is Already Over
By the time you're looking at an incident, the state that caused it is usually gone. Debugging production is forensics on a scene that's already been cleaned up — which is why what you captured while it was happening matters more than how hard you look afterward.
The Question You Didn't Instrument
In production you can only answer the questions you decided to measure in advance. The most useful metric is almost always the one someone added before anyone needed it — and the worst incidents are the ones where the data you'd want simply doesn't exist.
The Alerting Paradox
The more alerts a system sends, the less anyone pays attention to them. Past a threshold, adding alerts makes a system less observable, not more — because the alerts that matter drown in the ones that don't.
The Error Budget
Perfect reliability is the wrong goal. An error budget turns reliability into a number you can spend — and once it's a budget, the argument about whether to ship stops being a matter of opinion.
The Recovery Cost
How long a system takes to recover from a failure is as important as how often it fails. A system that fails rarely but recovers slowly can accumulate more total downtime than one that fails often but recovers fast.
The First Month
New systems tend to be most reliable in their first month of operation — not because they're less likely to fail, but because operators are more likely to be watching. Vigilance decays faster than systems do.
The Known Good State
In debugging and reliability work, the most valuable reference point is a known good state — what the system looked like when it was working. Systems that don't capture baselines lose the ability to detect the moment they drift away from one.
The Near Miss
Near-misses are higher-value reliability signals than actual incidents because they surface failure modes without the cost of actual failure. But most teams only run post-mortems on incidents that broke through, so near-miss signals evaporate before anyone learns from them.
Output-First Observability
Most monitoring is built around processes: did it run, did it error, did it use too much memory. Output-first observability flips that — it asks whether the thing that was supposed to be produced exists, is current, and is correct.
The Alert That Arrived Too Late
An alert that fires after a problem has been accumulating for weeks isn't a monitoring system — it's a postmortem trigger. The gap between when the failure started and when the alert fires is where the actual cost lives.
The Freshness Signal
Freshness is a property of output, not process. Knowing that something ran is not the same as knowing that what it produced is still current. That distinction is where stale-data bugs hide.
The Silent Accumulation
The most dangerous gaps are the ones that don't announce themselves. They accumulate quietly, invisible until suddenly the distance between where you are and where you should be is too wide to ignore.
What Monitoring Misses
Monitoring tells you what happened. It doesn't tell you what didn't happen. That asymmetry is where most silent failures hide.
One Failure Is Not an Incident
Alert thresholds exist for a reason. A monitoring system that wakes you up for a single transient error isn't protecting you — it's training you to ignore alerts.
Telemetry Without the SDK
OpenTelemetry SDKs bring a lot of weight. When you're in a constrained environment, you can get full observability by speaking the wire protocol directly.
Healthy Process, Empty Pipe
A system can run without errors and produce nothing at all. Those are different failure modes, and only one of them shows up in your uptime metrics.
The Log Nobody Reads
Why writing logs matters even when nobody checks them