A scheduled run on a system we maintain had been failing two of its nine jobs every morning. The summary said TIMEOUT, which sounds like something is stuck.
Nothing was stuck. Both jobs were working perfectly, and we were killing them two-thirds of the way through.
What had actually happened
The scheduler reads an estimate for each job and allows it three times that long before giving up. One job was listed at two minutes, another at three. So they were being cut off at six and nine.
Those estimates were accurate when they were written. At the time, the job scanned a watchlist of 39 companies. It now scans just over 500.
Nobody changed the estimate, because nobody thought of it as a setting. It was a note about how long something took — right when it was written, and never revisited.
The error pointed at the wrong thing
This is the part worth sitting with. The failure was reported as a timeout, and a timeout reads as “the job hung”. The obvious response is to go and look at the job.
We only found it because the logs showed the work in progress — ticker 215 of 503, then nothing. The job was fine. The number describing it was two years out of date.
An error that points at the wrong component is worse than no error. It sends you somewhere confidently.
The bit that made it stick
The timeout was never set directly. It was derived — three times an estimate written for a different purpose entirely.
That matters, because the obvious fix is to raise the timeout, and that would have worked. It would also have left the stale number sitting there feeding everything else that reads it: the “this will take about 47 minutes” line before a run, any planning done off the back of it, and the next thing anyone builds on top.
Fixing a symptom derived from a wrong fact leaves the wrong fact in place.
Where this lives in a normal business
Once you have seen the shape, it is everywhere:
- An approval threshold set years ago, so a limit that used to catch the exceptions now catches everything.
- A report timeout from when the table had a tenth of the rows, so the monthly export quietly returns partial data.
- A delivery cutoff set when there was one van.
- A batch size that fitted the old machine.
- A staffing ratio from before the product changed shape.
- An insurance limit, a credit limit, a file size limit — each correct on the day it was chosen.
None of these look like settings. They look like facts. That is exactly why they do not get reviewed.
What to do about it
- Write down when a number was set and what it was measured against. “2 minutes” is a fact with no shelf life. “2 minutes for 39 records, measured March 2024” expires by itself.
- Measure rather than assume, once. Our real figures were 14 and 23 minutes. We had been guessing at 2 and 3 for long enough that nobody questioned it.
- When something grows by an order of magnitude, go looking for the numbers that describe it. Growth does not break things on its own — it breaks the constants around them.
- Make failures say what they saw. “Timed out at 6 minutes, reached record 215 of 503” sends you to the right place. “TIMEOUT” sends you to the wrong one.
The honest version
The fix took four minutes once we understood it, and it had been failing every morning for a while before anyone looked properly, because the failure was plausible — scans time out sometimes, and the summary gave a reason that sounded like an answer.
That is the general case. These do not survive because they are hard to fix. They survive because the explanation on the screen is good enough to stop you asking.