Feeds

Lateral thought saves sizzling server

Game, set and crash

HP ProLiant Gen8: Integrated lifecycle automation

D'oh! I learned a long time ago that generating random numbers (really, truly random numbers) is a non-trivial exercise.

However, I completely failed to apply that computer science lesson to the real world of computing and continued to believe that events in the Newtonian world could happen without a cause. Such a belief system is not usually dangerous but, when applied to solving computer problems, it can be a serious disadvantage.

I came to my senses after I had spent a great deal of time trying to track down an intermittent fault on a NetWare server. It crashed. Then it ran fine for days. And then it crashed again. And again. Apparently, at random.

We would get a call from the local supervisor Rosanne whenever it crashed. By the time we got there (it was offsite) the server would reboot as if nothing had happened and run like a dream.

Sometimes it would run for weeks, other times it crashed three days running. The only correlation we could spot was that the crashes were always during the day so in that sense it wasn't random but since days happen seven times a week, every week without fail, it wasn't really a great help in diagnosing the problem and curing it.

And during the day there was absolutely no correlation with load. We concluded that the server was crashing at random and started the process of swapping parts (at random!) to try to cure it.

Then, on one of our frequent visits, Rosanne said jokingly that we really had to fix the problem because it was ruining her social life. The server crashed every time she played tennis with her new boyfriend. By this time we were desperate to find any correlation between the crashes and real life so we rather startled her by resurrecting the Spanish inquisition.

Was she serious? Well, er... not every time but, yeah, her boyfriend had pulled her leg that she was setting off her pager on purpose to avoid losing games and she had realized that it did seem to happen all too frequently. How often did she play? Well, a couple of time a week, maybe; it depended.

On what? How did she decide to play? Well, both she and her boyfriend worked flexitime so whenever the weather was good, they booked a court and played a game. They made up the missing time by working an hour later in the evening.

Ace in the hole

So the server was crashing when the weather was good. OK, how do we define good weather in Scotland where this was all happening? It is good weather if the sun shines. What happens when the sun shines? The sky is bluer, there are fewer clouds, the humidity is probably lower... it gets hotter. Hmm.

Servers don't like heat. Where is the server? Sitting on a bench. In front of a south facing window - this was in the days before server rooms, when air conditioning was provided only for mainframes.

So, Rosanne plays tennis when the weather is good, the sun shines and it's cooking the server. The Newtonian world is back in balance, yin has a yang and effect does have a cause.

I am, I like to think, at least slightly wiser now. I learned from that particular lesson that saying: "It must be random" is another way of saying that I have yet to find the correlation. Worse than that, it's usually a cop out.®

Doh! Is Mark Whitehorn's look at the events, and lessons learned, that served him well during his computing career.

Eight steps to building an HP BladeSystem

More from The Register

next story
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
prev story

Whitepapers

Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.