Boss hands dunce's cap to chap who turned off disk monitor
What's the point of a server that can't tell you how it feels?
On-Call If it's Sunday, it must be time for On-Call, our regular look at the messes Reg readers are asked to clean up at night, on weekends or at other times when they should by rights be enjoying themselves.
This week, reader Oliver recalls the time when he worked among “a small but happy team of engineers” at “an IT support company that looked after a number of companies”.
“We all had worked in the industry long enough to know good practice from bad,” Oliver says, which was why the company had been able to score a football club as a client.
One weekend, Oliver says he “received an urgent phone call from the football club's account manager as their email was down”.
Join the dots, people: football clubs need their email on weekends more than the rest of us, so Oliver roared into action.
“I remote-accessed their exchange server to find that one of my co-workers had disabled the disk space alerts from our monitoring system and turned off the 'backpressure' check in Exchange.”
The result? “There was ZERO disk space left on the disk with the Exchange database and no way of clearing anything without input from the customers.”
Oliver was able to figure out that “the pagefile/virtual memory had been set to use this disk and was able to free 32GB this way”. With that sorted, the server was back in business.
Come Monday, two extra disks were ordered and the database split up into safer sections. But the chap who flicked the monitor to OFF didn't fare as well. He, Oliver says, “was shocked to find the Technical Director joining him on his next site visit to the club to show him how to set up an Exchange server properly”.
What have you been asked to do after hours? Tell me and you could find your own sensitively-anonymised tale in a future On-Call. ®
Sponsored: Becoming a Pragmatic Security Leader