Techie was bigged up by boss… only to cause mass Microsoft Exchange outage

Who, Me? Welcome once more to Who, Me?, our weekly column in which readers confess to their worst IT cock-ups.

This time, we meet “Freddie”, who must have a good supply of luck on his side, as he managed to not only get away with his error, but win praise for fixing it.

At the time, Freddie was a busy man. He worked for a managed service provider as the supervisor for all technical support staff, was the company's senior network engineer and its security specialist.

“As if wearing three hats wasn't enough, due to a shortage in employees, I was also made the lead network administrator for one of our bigger clients,” Freddie told us.

“What that boiled down to was doing all of the work I normally do, but with the occasional system or network task, seated Monday through Friday in this customer's network operations centre.”

One week, Freddie had to suffer through a cringe-inducing meeting in which his boss told the business manager and IT coordinator of the customer just how great he was.

Of course, with that sort of glowing endorsement, there was really only one thing for Freddie’s reputation to do: go downhill.

The blunder came a week later, when he was performing some maintenance on the customer’s Exchange environment.

“Conveniently for me, there was a de facto maintenance window every morning from 6:30am-7:00am,” he said. The maintenance he was carrying out wasn’t expected to cause any downtime, but he’d been asked to do it during this window anyway.

“I had just finished with five minutes to spare, but instead of hitting 'sign-out', I accidentally hit 'shut-down' on the primary server,” Freddie said.

Thinking that this wasn’t the worst thing in the world, Freddie hurriedly restarted the virtual machine.

“But, unbeknownst to me, my predecessor had incorrectly removed a server (or maybe servers) from the [database availability group],” Freddie said – which meant starting Exchange services took about 20 minutes.

Not long after the work day was supposed to start, Freddie was on the receiving end of a call from the same business manager.

"Everyone up here is getting a message that we can't log in to our email,” the chap said. “Help!”

“Oh really?” Freddie, thinking on his feet, replied. “Let me take a look, and I'll see if I can get that fixed... Give me maybe 10 minutes?"

Feeling a quiet confidence that his ruse was working, Freddie crossed his fingers and waited.

“Ten minutes later, Exchange was back online, and I called the business manager back, asking him to try it again.

“A moment later he replied, ‘Freddie! I don't know what you did to fix it, but it's working for everyone now! Great job!’”

Better yet, an hour or so later, Freddie’s boss at the managed service provider also called to praise his work.

“He’d received a call from the client telling him how quickly I was able to fix the issue,” Freddie said.

If this tale has reminded you of a time you took credit for fixing a fault you created, don’t keep quiet – tell Who, Me? and we might run your tale in the future.

And don’t forget, we’re on the hunt for spooky tales for a Halloween special of On Call, our column for tech support triumphs. ®