Code Doesn't Keep
Code that never runs isn't preserved — it holds references to an environment that keeps moving without it. The common path is continuously re-validated by reality; the rare path is a stopped clock nobody has looked at.
Operations
The Rare Path Gets Rarer
Every fix to the common path makes the exception path less frequent, less familiar, and less examined — so reliability work quietly concentrates your remaining risk in the code nobody has watched run.
You Can't Store Readiness
Execution is the only maintenance mechanism that reliably works, and nobody schedules it. That makes readiness a flow rather than a stock — which is why the only honest question about any part of a system is when it last actually ran.
Nobody Files a Ticket to Lose Access
Permission systems drift toward maximum grant, not because anyone decided to loosen them, but because only one direction of change has someone asking for it.
Only One Side Leaves a Record
Access lasts too long and arrives too wide for the same reason: granting produces an artifact and needing produces nothing, so every correction has to argue from silence.
The Role Fits Nobody
Access expires too slowly, but it also arrives too wide. Roles grow to the union of everything their members have ever needed, and everyone holds the whole union.
Can't Reproduce Is a Measurement
Failing to reproduce a bug tells you something real. It just doesn't tell you the thing most people close the ticket believing it said.
The Only Failure That Counts
Both halves of reproduction — succeeding and failing — quietly replace the reporter's failure with yours. Only one of the two is the bug.
The Repro Is a Model
Getting a bug to reproduce feels like the end of the investigation. It's the point where you quietly substitute your version of the failure for theirs.
The Decision Nobody Made
Most of a system's shape was never chosen. It's the fossilized remains of whatever was expedient the first time, and it constrains everything downstream as firmly as if someone had decided it on purpose.
The Moment It Stops Being Provisional
Accreted structure isn't only an archaeology problem. It's being created right now, and the moment a temporary shape becomes permanent is observable while it's happening.
The Question That Doesn't Decay
Provenance is a proxy. What actually holds a system's shape in place isn't anyone's reasoning — it's the count of things standing on it, and that you can measure at any time.
One Choice, Two Costs
Picking a metric and starving the work it can't see aren't two separate problems. They're the same act, viewed from either side.
The Metric Stopped Being the Thing
A proxy measurement is only honest for as long as nobody is optimizing against it yet. The moment it becomes the target, it starts drifting away from the thing it was supposed to represent.
The Work That Never Shows Up
Every measurement system creates two categories: work that counts and work that doesn't. The second category is where prevention lives, and it starves quietly.
Someone Is Always Waiting on You
The fastest responder on a team accumulates dependency without anyone deciding to give it to them. Being good at unblocking others is how you become the block.
The Test Is Whether You Can Leave
You can't fix a dependency bottleneck by working harder inside it. The only honest measure of how well a team is structured is what happens during the week nobody can reach you.
You Are in a Queue You Cannot See
The person you asked is holding eleven other requests. You can't see them, so you assume you're the only one — and that assumption is what makes the queue grow.
Assume It's Load-Bearing
You inherited code with no explanation attached. The safe default is not caution, and it is not confidence — it is finding out.
The Handoff Is Not an Event
Transfer documents fail because they are written by someone who has already stopped being the owner. The fix is to stop treating the transfer as the moment.
What the Handoff Drops
Work changes hands constantly, and every transfer loses something nobody wrote down because nobody knew it was load-bearing.
The List That Got Too Long
Every list view is designed against twenty rows and lived in at twenty thousand. Finding things is a feature, and it gets scoped as decoration.
The View Somebody Saved
Once people can filter and search, they start building. What they build becomes shared infrastructure nobody planned to maintain.
They Type What They Remember
Search inside a product fails on the queries people actually make: partial names, misspellings, and the one detail they happen to recall.
Let Them Take It With Them
Export is the feature nobody demos and everybody evaluates. A product that makes leaving hard is not one people commit to.
The Import Is the First Impression
The first thing a new customer does with your product is hand it a file of their real data. Whatever happens next is what they learn about the software.
Every Notification Spends Attention
Notifications get added one feature at a time, each individually justified. Nobody owns the total, and the total is what determines whether any of them get read.
Let Them Turn It Off
Notification preferences look like a courtesy. They're actually the mechanism that keeps the channel usable — and the coarser they are, the more people mute everything.
The Message That Arrived Twice
Notifications are deliveries to systems you don't control, on paths that can retry. The duplicate that reaches a customer is more visible than almost any other bug.
Automate the Fix or Remove the Need
A recurring manual repair is a bug report written in someone's calendar. Building a faster way to perform it is progress; not needing it is the actual goal.
The Admin Tool Is Production
Internal tools get built quickly, reviewed lightly, and given more power than anything customers can touch. They deserve the standards the customer-facing paths get.
The Tool Support Built in a Spreadsheet
When the internal tooling doesn't cover a case, nobody files a ticket. They invent a workaround, and the workaround becomes permanent infrastructure nobody planned.
A Rate Limit Is a Message
Limits get added to protect the system, then serve as the product's only communication about how much use is acceptable. Most say it badly.
Fairness Is Something You Build
Shared capacity is first-come, first-served by default, which means the heaviest user sets everyone else's experience. Nothing about that is automatic to fix.
The Limit You Never Raised
Caps are set once, at a moment that quickly stops resembling the present. The ones that stay put become invisible ceilings on what customers can do with your product.
A Timestamp Without a Zone Is a Guess
Storing when something happened is easy. Storing it in a way that still means the same thing in another country, six months later, is where systems quietly go wrong.
The Scheduled Job That Ran Twice
Recurring work looks simple until you ask what happens when a run is late, overlaps the next one, or fires on a machine that thinks it's a different hour.
Nobody Is Sure This Is Unused
Code accumulates not because anyone wants it, but because removing it requires certainty nobody has. The fix is making that certainty cheap to obtain.
Removal Needs an Owner Too
Everything that gets built has someone who wanted it. Almost nothing that should be removed has anyone whose job it is to notice.
The Feature Two Customers Use
The hardest things to remove aren't unused — they're barely used. Someone real depends on them, and that's enough to keep them alive indefinitely by default.
Hiding the Button Is Not a Check
Where a permission is enforced matters more than how it's modeled. Interfaces hide options; only the layer that touches data can actually deny anything.
Permissions Are a Product Decision
Access control gets treated as plumbing and built by whoever drew the short straw. It's actually a description of how your customers' organizations work.
Someone Has to Be Able to Answer This
"Who can see this record, and why?" is the question a permission system exists to answer. If nobody can answer it without reading code, the system has already failed.
Know What Leaving Would Cost
Switching costs accumulate quietly from the day you integrate. The useful question isn't whether you're locked in — it's whether you know the number.
They Changed It and You Didn't Deploy
Your code is identical to yesterday's and the behavior is different, because the change happened on the other side of an integration you don't control.
You Inherited Their Failure Modes
Integrating a third-party service imports more than its features. It imports their latency, their outage windows, their rate limits, and their idea of what an error means.
Tell the User the Work Is Pending
When work moves to a queue, the interface usually keeps claiming it's done. Closing the loop means the product tells the truth about what has actually happened yet.
The Message That Can Never Succeed
Most queue failures are transient and retrying fixes them. The interesting case is the message that will fail identically forever, and what your system does when it meets one.
The Queue Is Where the Work Hides
Moving work to a background queue makes the request fast. It doesn't make the work smaller — it moves it somewhere with fewer people watching.
The ALTER Statement Is the Easy Part
Changing a table's definition takes one line. Getting millions of existing rows into the new shape, while the system keeps running, is the actual project.
The Schema Outlives the Code
Applications get rewritten, frameworks get replaced, services get split apart. The data usually survives all of it, which is why the schema is the most expensive decision in the system.
Follow One Request All the Way Through
Well-written, well-rationed log lines still fail if you can't assemble them into one story. The capability that makes logs worth keeping is being able to trace a single request end to end.
Logging Everything Is Not a Strategy
"Log it just in case" feels like insurance. What it actually buys is a haystack, a storage bill, and a search that times out during the incident you bought it for.
Logs Are Written for the Wrong Reader
Most log lines are written by someone who already knows what the code does, for a reader who doesn't and won't be able to ask.
Config Changes Are Deploys
A config edit can change production behavior as completely as a code change can, and in most places it does so with none of the review, testing, or staged rollout that code gets.
Every Setting Is a Deferred Decision
Most config options exist because someone couldn't decide, or didn't want to. The knob ships, the decision never gets made, and everyone downstream inherits the question.
Nobody Knows What the Config Actually Is
The value a service runs with is assembled from defaults, files, environment variables, a remote store, and per-tenant overrides. Very few systems can tell you what won.
Every Alert Is a Claim About a Person
An alert isn't a statement about a metric. It's an assertion that a specific human should stop what they're doing and act — and most alerts were never designed to earn that.
The Response Is Part of the Alert
An alert that fires correctly and leaves the person receiving it with no idea what to do has done half a job. The response isn't downstream of the alert — it's the reason the alert exists.
You Alert on the Failures You've Already Had
Alert rules accumulate one incident at a time, which means your coverage is a map of your history — not of the ways your system can actually break.
A Rollback Is Not a Time Machine
Rolling back a deploy feels like undoing it. Mostly it undoes the code. Everything the bad code already did to your data, your queues, and your downstream systems is still there, waiting.
Both Versions Are Running
A deploy isn't a moment when the old code becomes the new code. It's a window where both are live at once, reading and writing the same data — and most deploy surprises live inside that window.
Ship the Code, Not the Change
Deploying code and turning on new behavior are two separate decisions that most teams make simultaneously by default. Separating them is what turns an irreversible deploy into a reversible one.
A Retry Is a Second Request
Retrying a failed operation feels like giving it another chance to succeed. What it actually does is ask a question nobody thought to answer: what happens if the first attempt worked and the failure was just in hearing about it?
Your Retries Are Someone Else's Load
A retry looks like local resilience — my request failed, I'll try again. At scale it's a decision about how much extra load to send a system that may already be struggling, made by every caller independently and at once.
Cache the Boring Way First
Clever caching strategies solve problems most systems don't have yet, and create ones most systems can't afford. The boring cache — short TTL, simple key, easy to reason about — is usually the right amount of cleverness.
The Cache Key Is the Spec
A cache key is a claim about what makes two requests the same. Get that claim slightly wrong and the cache doesn't fail loudly — it just quietly serves the wrong answer to someone.
Build for Now, Instrument for Later
You can't design for a scale you don't have yet without paying for flexibility you may never use. What you can do is build honestly for today and leave yourself a way to notice the exact moment today stops being enough.
The Assumption That Worked at Ten
Every system is built on assumptions that were true at the scale it was built for. Growth doesn't announce which ones stopped holding — it just quietly waits for you to find out the expensive way.
The Shape Hiding in the Loop
Code that looks like it does one pass over the data can secretly do one pass per item — a hidden multiplication that's invisible at small size and unmissable once the input grows.
A Test Proves One Thing
A green test suite feels like a broad statement about your code's health. It's actually a narrow one: these specific inputs produced these specific outputs, today. Confusing the two is where false confidence comes from.
Tested Is Not the Same as Verified
A test that runs and passes tells you the code did what the test checked. It doesn't tell you the test checked the right thing. That second question is easy to skip and expensive to skip.
Deprecate Like You Mean It
A deprecation notice that nobody acts on isn't a warning, it's decoration. The gap between marking something deprecated and actually being able to remove it is where most interfaces quietly calcify.
The Contract Is What They Rely On
You decide what your interface promises. Your users decide what they depend on. When those two differ — and they always do — the second one is the real contract.
Version the Promise, Not the Code
A version number is supposed to tell callers something. Too often it just tells them the code changed — which they already knew, and which doesn't help them decide whether to worry.
Every Error Is a Design Decision
Error handling gets treated as the cleanup after the real work — the branch you fill in to make the compiler happy. But what a system does when something goes wrong is part of what the system is.
The Failure You Planned For
A failure you anticipated is an inconvenience. The same failure unanticipated is an incident. The difference isn't in the event — it's in whether the system had somewhere to put it.
Write the Error for the Person Reading It
An error message is written once, in a moment of frustration, by someone who already knows what went wrong. It's then read by people who don't — often at their worst moment, with no other information to go on.
One Wrong Default, Times Everyone
A bug in a rarely-used option affects the people who chose it. A bug in the default affects everyone who didn't choose anything — which is usually almost everyone. The blast radius of a mistake tracks how many people never had to opt in.
The Default Is a Decision
Defaults feel like the absence of a choice — the value nobody had to set. In practice they're the choice most users will live with, made once by someone who won't be there to see the consequences.
The Setting Nobody Should Need
Adding a setting can be a way of avoiding a decision — shipping both answers instead of finding the right one. Sometimes that's respect for real variation. Sometimes it's an unresolved argument, permanently installed.
Shorten the Loop Before You Optimize It
When work feels slow, the instinct is to get better at the steps inside the loop. Usually the bigger win is making the loop itself shorter — so being wrong stops costing so much.
The Code You Can't Loop On
Some code has no fast feedback loop at all — you can't easily run it, watch it, or reproduce its failures. That code doesn't just move slowly. It resists being understood, and the slowness compounds.
The Loop Is the Unit of Speed
How fast you build isn't set by how fast you type. It's set by how quickly you can go around the loop of making a change, seeing what it did, and learning from it. That cycle time is the real speed.
Code Is a Liability
The instinct is to count code as an asset — look how much we built. But the asset is the behavior; the code is what you pay to keep it. More lines doing the same job is more liability for the same value.
The Flexibility You Didn't Need
Building for an imagined future feels like foresight. Usually it's a bet against odds you'd never take if you saw them clearly: pay the cost of flexibility now, on a guess about needs that mostly never arrive.
The Simplest Thing That Works
If code is a liability and speculative flexibility is a bad bet, the discipline that follows is to build the simplest thing that solves today's real problem. The hard part is telling simple from naive.
The Comment That Should Have Been Code
The urge to write a comment explaining what a piece of code does is usually a signal, not a solution. Most of the time the honest fix is to make the code say it — and a comment is what you write only when the code can't.
The Cost of Surprise
Code that works but behaves unexpectedly still charges a tax: every reader has to stop and verify the thing they assumed. Consistency isn't aesthetic tidiness — it's what lets people trust their assumptions and move on.
The Name Is the Understanding
Struggling to name something isn't a vocabulary problem. It's the code telling you that you don't yet understand the thing you're building — and a good name is what understanding looks like once you do.
Fast Enough Is a Real Number
Performance has a target, and the target is almost never 'as fast as possible.' It's a specific threshold tied to what a human perceives or a system requires — and knowing that number is what tells you when to stop.
The Shape Beats the Constant
Once you've measured, most real speedups don't come from making the code faster. They come from making the code do less — changing how the work grows with the input, not shaving the cost of each step.
You Are Guessing About Speed
Performance intuition is wrong often enough to be dangerous. The slow part is rarely where it feels like it should be — and the only way to know is to measure the specific system, not reason about it.
The Deadline Doesn't Change the Work
A deadline sets when you want something, not how much work it is. When the two collide, the honest levers are few — and the popular ones, adding people and working harder, mostly make it worse.
The Estimate Is a Distribution
When you give a task a single number, you've hidden the only thing that mattered: the spread. A three-day estimate that's really 'two to fifteen' isn't a smaller version of the same answer — it's a different kind of answer.
The Second Ninety Percent
The old joke — the first 90% of the work takes 90% of the time, and the last 10% takes the other 90% — isn't cynicism. It's a precise description of where estimates go to die: the unglamorous finishing that no one pictures.
Coverage Is Not Confidence
A coverage number tells you which lines ran during the tests. It says nothing about whether anything was actually checked — and the gap between those two is where teams get a false sense of safety.
The Mock That Agreed With You
A mock replaces a real dependency with your belief about how it behaves. When the belief is wrong, the test passes and production fails — because you tested the version of the world in your head, not the one that exists.
The Test That Broke for the Wrong Reason
A test that fails when you refactor working code isn't protecting you — it's charging you. And the real damage isn't the wasted hour; it's that the suite slowly teaches people that failures don't mean anything.
A Dependency Is a Standing Obligation
Adding a library is priced as a one-time decision — an afternoon saved. It's really a subscription: upgrades, CVEs, breaking changes, and the day it's abandoned. The install is the cheapest moment you'll ever have with it.
The Exit You Never Designed
Whether a dependency is cheap or ruinous mostly comes down to one thing decided at integration time: can you leave? That's not a property of the vendor. It's a property of how far its concepts spread into your code.
Their Uptime Is Your Uptime
A library is an obligation you carry. A service you call at runtime is stronger than that: you've adopted its availability as a ceiling on your own, and the arithmetic of that compounds faster than anyone expects.
The Code Outlives the Reason
Code persists perfectly; the reasoning behind it evaporates. Which is why so much of a mature codebase is lines nobody dares touch — not because they're wrong, but because nobody remembers what they were for.
The Rewrite Is a Knowledge Bet
Rewriting feels like replacing bad code with good code. It's really replacing accumulated knowledge with a fresh guess — and the ugliness you're removing is often where that knowledge is stored.
You Can't Document a Mental Model
Documentation transfers facts. It doesn't transfer the model that makes those facts usable — the sense of how the system moves. That's why the person who's been there three years still answers questions the wiki technically already answers.
Make the Bad State Impossible
Checking for invalid states at runtime means remembering to check everywhere. Shaping your data so the invalid state can't be expressed at all means you only have to be right once — at the type, not at every call site.
State Is the Hard Part
Computation is mostly easy; the hard part of software is state — the accumulated memory of everything that happened before. Most bugs aren't wrong logic, they're the system being in a combination of states nobody pictured.
Two Copies of the Truth
The moment a fact lives in two places, you've taken on a job you'll eventually fail at: keeping them equal. Most 'impossible' bugs are just two copies of something that were supposed to agree and quietly stopped.
Draw the Line Where It Changes
A module boundary is a bet about what will change together. Draw it along the axis of change and edits stay local; draw it along surface resemblance and every change cuts across every module.
The Abstraction That Leaks
Every useful abstraction hides the layer beneath it — until the day it can't. The ones that serve you longest aren't the ones that hide the most, but the ones that fail honestly when the thing underneath breaks through.
Wait for the Third Case
The wrong abstraction is more expensive than duplication, because it's harder to reverse. Which is an argument for waiting — abstracting on the third occurrence, not the first, when you finally know which parts actually vary.
Normal Is a Measurement
A number from production means nothing until you know what that number usually is. The hardest part of observability isn't collecting metrics — it's knowing what normal looks like, because 'bad' is defined entirely by contrast with a baseline you had to measure first.
The Incident Is Already Over
By the time you're looking at an incident, the state that caused it is usually gone. Debugging production is forensics on a scene that's already been cleaned up — which is why what you captured while it was happening matters more than how hard you look afterward.
The Question You Didn't Instrument
In production you can only answer the questions you decided to measure in advance. The most useful metric is almost always the one someone added before anyone needed it — and the worst incidents are the ones where the data you'd want simply doesn't exist.
The Expand-Contract Migration
A schema change and a code change can't deploy at the same instant. Expand-contract accepts that and makes the intermediate state — where both old and new must work — the thing you design for.
The Flag That Outlived Its Change
Feature flags are what make progressive rollout and safe migration possible. They're also the debt those techniques quietly accumulate — and the flag you never delete is the one that decides your incident for you.
The Rollout Is the Test
No test environment fully reproduces production. That's not a gap to close — it's a fact to design around, which means the rollout itself has to be the final test, run against real traffic in a way that limits what a failure costs.
The Accidental Interface
The interfaces you have to keep stable aren't just the ones you designed. Anything observable becomes something someone depends on — including the details you never meant to promise.
The Compatibility Contract
The moment another system depends on your interface, the interface stops being yours to change freely. Backward compatibility is the contract you signed without reading it, and breaking it breaks things you can't see.
The Deprecation That Never Ends
Marking something deprecated is easy. Removing it is the hard part, and most deprecations never get there — they just accumulate, and the old thing runs forever alongside the new one.
The At-Least-Once Default
Most messaging systems promise to deliver each message at least once, not exactly once. The gap between what you assumed and what the system actually guarantees is where the duplicate-processing bugs live.
The Ordering Assumption
Messages arrive in the order they were sent — until they don't. Assuming global ordering in a distributed system is one of those beliefs that holds in testing and breaks in production, quietly, in ways that are hard to trace.
The Alerting Paradox
The more alerts a system sends, the less anyone pays attention to them. Past a threshold, adding alerts makes a system less observable, not more — because the alerts that matter drown in the ones that don't.
The Error Budget
Perfect reliability is the wrong goal. An error budget turns reliability into a number you can spend — and once it's a budget, the argument about whether to ship stops being a matter of opinion.
The Graceful Degradation Default
When a dependency fails, a system has two options: fail with it, or degrade around it. Most systems fail with it — not because degrading is impossible, but because nobody decided in advance what the degraded state should be.
The Blast Radius
When a system fails, how much else fails with it? The blast radius of a failure is a design property, not an accident. Systems that fail with a small blast radius are easier to recover from, easier to debug, and less expensive to operate.
The Recovery Cost
How long a system takes to recover from a failure is as important as how often it fails. A system that fails rarely but recovers slowly can accumulate more total downtime than one that fails often but recovers fast.
The Runbook Gap
A runbook written the day after an incident captures what you wish you'd known. A runbook written six months later captures what you remember. The gap between those two is where the operational knowledge goes.
The First Month
New systems tend to be most reliable in their first month of operation — not because they're less likely to fail, but because operators are more likely to be watching. Vigilance decays faster than systems do.
The Near Miss
Near-misses are higher-value reliability signals than actual incidents because they surface failure modes without the cost of actual failure. But most teams only run post-mortems on incidents that broke through, so near-miss signals evaporate before anyone learns from them.
Output-First Observability
Most monitoring is built around processes: did it run, did it error, did it use too much memory. Output-first observability flips that — it asks whether the thing that was supposed to be produced exists, is current, and is correct.
The Alert That Arrived Too Late
An alert that fires after a problem has been accumulating for weeks isn't a monitoring system — it's a postmortem trigger. The gap between when the failure started and when the alert fires is where the actual cost lives.
The Recovery Window
When a system has been silent for weeks, recovery isn't just restoration — it's reconstruction. How you handle the gap matters as much as fixing the underlying failure.
The Failure That Stays Quiet
The dangerous failures aren't the ones that throw errors. They're the ones that fail silently, leave no alarm, and only surface as drift you notice later. The defense is building routines that verify state instead of trusting the last run.
The Discipline of the Boring Check
Running the same health checks when everything is fine feels like wasted motion. It isn't. The boring check that almost always passes is what makes the rare failure visible the moment it happens.
One Failure Is Not an Incident
Alert thresholds exist for a reason. A monitoring system that wakes you up for a single transient error isn't protecting you — it's training you to ignore alerts.