The Scheduled Job That Ran Twice
The two pieces before this one were about representing time and about not trusting clocks to agree. Scheduled work sits on top of both, which is why it fails in ways that surprise people who thought they’d written a cron line. “Run this every hour” sounds like a complete specification, and it leaves out nearly everything that determines whether the thing works: what happens if a run takes more than an hour, if the machine was down when it should have fired, if two machines both think they’re responsible, if the hour in question doesn’t exist because the clock jumped.
Overlap is the first thing that bites. A job that usually takes two minutes will one day take seventy, and by default the scheduler starts the next one anyway — so now two copies are processing the same records with no coordination, which is the concurrency problem from the queues thread arriving on a timer. The fix is deciding explicitly: skip this run if the previous is still going, or queue it, or allow overlap because the work is genuinely independent. All three are defensible. The default — whatever the scheduler happens to do — is the only choice that wasn’t made deliberately.
Then there’s the missed run. The machine was restarting, the deploy was mid-flight, the scheduler itself was down for ten minutes. When it comes back, should the skipped execution happen late, or be abandoned? For a report, running late is fine and probably desirable. For something that acts on a time window — “charge everyone whose trial ended in the last hour” — running late is fine only if the job computes its window from the data rather than from “now minus one hour,” which is the difference between a job that’s robust to lateness and one that silently skips a slice of records whenever it’s delayed.
Duplicate execution deserves its own attention because it’s the failure that produces the worst outcomes and the least evidence. Two schedulers after a failover, a retry after a timeout that actually succeeded, a DST repeat — and the job runs twice. If it’s idempotent, nothing happens; if it isn’t, you’ve sent the emails twice or double-applied the charges, and nothing in the logs says anything went wrong, because from each run’s perspective nothing did. This is exactly the retries argument again, arriving through the scheduler: the honest question isn’t how often the job runs, it’s what happens if it runs twice at once.
Which is the thread’s close. Time is treated as a background fact — the clock is right, the schedule fires, an hour is an hour — and it’s actually a set of assumptions the system depends on and rarely writes down. Store instants unambiguously. Measure durations with something monotonic. Decide what “on schedule” means when reality doesn’t cooperate. None of it is difficult; all of it is invisible until the night the clock changes, the failover happens, or the job that’s run cleanly for two years finally runs long.