Output-First Observability
The standard mental model for system health is process-centric: is the service up, is the job running, is the error rate within bounds. These are useful signals, but they share a structural weakness — they measure the activity of the system rather than the output of the system. A process can run continuously, log no errors, and consume normal resources while producing stale, incomplete, or subtly wrong output. Process health and output health are correlated, but not the same thing.
Output-first observability starts from a different question: what was this system supposed to produce, and does that thing exist in the right state right now? For a periodic job, the output is whatever it writes — a database table, a file, an index. The question isn’t whether the job ran. It’s whether the table was written recently, the file has the right format, the index covers the expected document set. For a cache, the output is the cached value. The question isn’t whether the cache was hit. It’s whether the cached value is fresh enough to be useful. For an API, the output is the response. The question isn’t whether the server returned 200. It’s whether the response body contains what it’s supposed to contain.
This reframe has several practical implications. First, the instrumentation lives at the output layer, not the process layer. You add a last_updated timestamp to the table, a freshness field to the cache, a schema validator to the API response. The monitoring check reads those fields and asks whether they’re in the expected state. This is often more work to set up than process-level monitoring, because you have to think carefully about what “correct output” looks like for each system. But the payoff is that you catch failure modes the process view can’t see — the job that runs but writes nothing, the cache that refreshes but writes a malformed value, the API that returns 200 but with an empty body.
Second, the alerting thresholds become more meaningful. A threshold like “error rate > 1%” requires calibration — what’s normal, what’s elevated, what indicates a real problem? A threshold like “last updated more than 4 hours ago” for a system that’s supposed to update hourly has an obvious correct answer. It’s either within the expected freshness window or it isn’t. The signal is less noisy and easier to act on.
Third, recovery is clearer. When a process-level alert fires, the first diagnostic question is always “what was it doing, and did it produce anything?” When an output-level alert fires, that question is already answered — you know the output is missing, stale, or wrong, and the investigation can start there rather than from the process logs. The failed state is better specified, which makes the remediation path shorter.
The shift is simple to describe and genuinely hard to apply uniformly. It requires knowing, for every system that matters, what its output looks like when it’s healthy. That knowledge isn’t always documented and sometimes isn’t obvious. But working through it — asking “what should this produce, and how would I know if it didn’t” for each system — tends to surface the monitoring gaps that matter, and those are the gaps where silent failures live.