'Can you just pop in to the office and hit the power button?' 'Not really... the G8 is on'
Not a great day for the n00b to power off the servers in a remote location
Who, Me? Monday, bloody Monday. If you haven't quite recovered from the weekend, just thank your lucky stars there aren’t any major global diplomatic events to tangle with today.
Unlike this week’s IT sinner, “Peter”, who writes in to El Reg’s regular confessional column Who, Me? to get an old incident off his chest.
At the time in question, Peter was in one of his first jobs out of school. “I was one of two local IT guys in an office of 100 people with a server environment running Novell Servers with Lotus Notes,” he said.
After paying his dues for a few years, Peter was trusted to manage a couple of the remote offices when that local office IT team was away.
On one such occasion, Peter discovered that the Arcserv backup application in the Seattle, Washington office wasn’t working correctly.
He explained that, like all good techies, he was "following the IT mantra of 'turn it off and on'..." – but since he was working remotely with no out-of-band console access, it was a reboot instead.
"I typed in the
reboot now option on the Novell console and it said,
Other servers are connected to the server. Do you want to shut down instead?."
As it was a file server, people had open files on their desktop – but that subconscious knowledge wasn’t enough to stop Peter's reflexes.
“I instinctively hit
Yes,”… then immediately went ‘No, no, no!’”
Ah yes, poor old Peter had just powered off a server in a remote office on a Saturday evening with no way to power it on, apart from calling the local IT staffer in the city.
“I sheepishly explained what had happened and asked him if he could go down to office and hit the power button,” Peter said – but it was not to be his day.
“His voice went up an octave: ‘Do you realize what is happening in Seattle right now?!’,” Peter recalled. Trying to keep things light, our man attempted a joke, replying: “It’s raining?”
You wish, Pete. Because the local man responded that the G8 Summit was in progress, there were riots in the city and the downtown core was shut down.
“Well, that complicates things a bit now, doesn't it?” Peter drily observed to us in the present day.
Nonetheless, the valiant Seattle-ite offered to head out into the fray and see if he could get to the office. He called back a couple of hours later to tell Peter the power was on – and that he owed him a beer.
Early experiment in mass email ends with mad dash across office to unplug mail gatewayREAD MORE
"Thankfully, when I logged in, the arcserv program was again functioning and I was able to do the backups," Peter said.
That night, he wrote to head office to explain the cockup, apologise and name-check his knight in shining armour, who was rightly thanked by the boss for going in on a Saturday night.
But the story doesn’t quite end there – because Peter later found out that the Seattle support man had kept a few details of his trip under his hat...
"He said that he hit a road block on the way to the office, explained to the officer what he was needing to do and where, and they took pity on him and drove him down to office in cop car.
"They waited outside for him to run in, hit the power switch and run back down and took him back to perimeter."
And while the cockup didn’t cost Peter his job, he did have to fork out for beer every time he visited that office.
Have you ever sent a colleague into a riot to clean up your mess? Tell Who, Me? what mistakes still have you reaching for the wallet in the pub on a Friday by sending us an email, here. ®