There’s a comfortable assumption buried in a lot of distributed systems code: that events happen in the order they were produced. You send message A, then message B, and the consumer sees A then B. It feels like it should be true — you did them in that order, after all — and in local testing it usually is, because there’s one producer, one path, and no contention. Then the system scales out, and the assumption quietly stops holding, and you get bugs where an update lands before the create it depends on, or a delete arrives before the thing it’s deleting.

The reason ordering breaks is that distributed systems have multiple paths, and different paths have different latencies. Two messages sent a millisecond apart can take different routes — different partitions, different network hops, different consumer instances — and arrive in either order. A message that fails and gets retried arrives after messages that were sent later but succeeded on the first try. A system that scales consumers horizontally has several workers pulling from the same source, and there’s no guarantee the one that grabbed the earlier message finishes before the one that grabbed the later message. Global ordering across a distributed system isn’t a mild simplification of reality; it’s a property that mostly doesn’t exist unless you specifically build for it, and building for it is expensive.

Most systems that do provide ordering provide it within a partition, not globally. A message queue might guarantee that messages with the same key are delivered in order, because they go to the same partition and the same consumer — but messages with different keys have no ordering relationship at all. This is usually the right tradeoff, because it gives you ordering exactly where you tend to need it (all the events for a single entity, in order) without the cost of a global sequence. But it means the ordering guarantee is narrower than “everything in order,” and code that assumes the broader guarantee will break on the events that fall outside the partition.

The defense is to stop depending on arrival order for correctness and start depending on something in the data itself. If each event carries a version number or a timestamp from its source, the consumer can order events by that field instead of by when they happened to arrive, discarding an update whose version is older than what it has already applied. This makes the system tolerant of reordering: a stale event that arrives late is recognized as stale and ignored, rather than being applied on top of newer data and corrupting it. The source-assigned version becomes the authority on order, and the unreliable arrival sequence stops being something correctness rests on.

This is the same move that idempotency and at-least-once handling require, pointed at a third failure mode. In each case the pattern is identical: the network gives you a weaker guarantee than your code wants to assume — it might deliver twice, it might deliver out of order, it might deliver late — and the fix is to stop trusting the network’s behavior and put the authority somewhere you control. An idempotency key for duplication, a processed-ID record for redelivery, a version field for ordering. The underlying discipline is one thing: never let correctness depend on a property the system doesn’t actually guarantee, and when you catch yourself assuming one, move the authority into the data.