The Blast Radius
Every system failure has a blast radius: the set of other systems, users, and processes that are affected when it goes down. A database failure might take down three services that depend on it. A shared authentication service failure might make every application that uses it inaccessible. A message queue failure might cause every consumer to back up simultaneously. The blast radius is determined by the system’s dependency structure, and that structure is a design choice.
Blast radius is worth reasoning about explicitly during system design, not just after incidents. The question “what else fails if this fails?” has an answer for every component, and that answer is often larger than it looks at first. Dependencies accumulate over time: a component that started with one upstream consumer acquires three more as other teams find it useful. Each new consumer extends the blast radius without anyone explicitly deciding that’s acceptable. The blast radius of the original component is now a product of its organizational adoption history, which is a fragile property to have.
The most direct way to limit blast radius is to limit dependencies. A service with one upstream consumer has a smaller blast radius than one with ten, all else being equal. This is a reason to be deliberately conservative about adding dependencies — not because the dependencies are individually bad, but because each one extends the blast radius of whatever you’re depending on and makes you part of the blast radius of everything you’re depending on. The dependency graph is a failure propagation graph, and it compounds.
Where dependencies can’t be eliminated, they can be made more graceful. Caches that serve stale data rather than returning errors when the source is down. Circuit breakers that stop requests to a failing dependency rather than letting them queue up and cascade. Feature flags that disable the parts of a product that depend on a failing system while the rest continues to work. These patterns don’t eliminate the blast radius, but they convert a hard failure into a degraded state — and degraded is often much better than down.
The last consideration is isolation. Services that share infrastructure — the same database server, the same cache cluster, the same message queue — have coupled blast radii. A problem in the shared infrastructure takes all of them down together. Services that have their own infrastructure have independent blast radii. The isolation costs more to operate but limits the damage when something goes wrong. The right tradeoff depends on the specific services and the acceptable risk, but it’s worth making the tradeoff deliberately rather than inheriting whatever isolation level the infrastructure happened to provide.