Lateral thought saves sizzling server
Game, set and crash
D'oh! I learned a long time ago that generating random numbers (really, truly random numbers) is a non-trivial exercise.
However, I completely failed to apply that computer science lesson to the real world of computing and continued to believe that events in the Newtonian world could happen without a cause. Such a belief system is not usually dangerous but, when applied to solving computer problems, it can be a serious disadvantage.
I came to my senses after I had spent a great deal of time trying to track down an intermittent fault on a NetWare server. It crashed. Then it ran fine for days. And then it crashed again. And again. Apparently, at random.
We would get a call from the local supervisor Rosanne whenever it crashed. By the time we got there (it was offsite) the server would reboot as if nothing had happened and run like a dream.
Sometimes it would run for weeks, other times it crashed three days running. The only correlation we could spot was that the crashes were always during the day so in that sense it wasn't random but since days happen seven times a week, every week without fail, it wasn't really a great help in diagnosing the problem and curing it.
And during the day there was absolutely no correlation with load. We concluded that the server was crashing at random and started the process of swapping parts (at random!) to try to cure it.
Then, on one of our frequent visits, Rosanne said jokingly that we really had to fix the problem because it was ruining her social life. The server crashed every time she played tennis with her new boyfriend. By this time we were desperate to find any correlation between the crashes and real life so we rather startled her by resurrecting the Spanish inquisition.
Was she serious? Well, er... not every time but, yeah, her boyfriend had pulled her leg that she was setting off her pager on purpose to avoid losing games and she had realized that it did seem to happen all too frequently. How often did she play? Well, a couple of time a week, maybe; it depended.
On what? How did she decide to play? Well, both she and her boyfriend worked flexitime so whenever the weather was good, they booked a court and played a game. They made up the missing time by working an hour later in the evening.
Ace in the hole
So the server was crashing when the weather was good. OK, how do we define good weather in Scotland where this was all happening? It is good weather if the sun shines. What happens when the sun shines? The sky is bluer, there are fewer clouds, the humidity is probably lower... it gets hotter. Hmm.
Servers don't like heat. Where is the server? Sitting on a bench. In front of a south facing window - this was in the days before server rooms, when air conditioning was provided only for mainframes.
So, Rosanne plays tennis when the weather is good, the sun shines and it's cooking the server. The Newtonian world is back in balance, yin has a yang and effect does have a cause.
I am, I like to think, at least slightly wiser now. I learned from that particular lesson that saying: "It must be random" is another way of saying that I have yet to find the correlation. Worse than that, it's usually a cop out.®
Doh! Is Mark Whitehorn's look at the events, and lessons learned, that served him well during his computing career.