Sysadmin held a rack of servers off the ground for 15 mins, crashed ISP when he put them down

Who, me? Welcome to the ninth edition of Who, me? In case you’ve come late to the party, it’s The Register’s Monday column featuring readers’ tales of stuffing things up.

This week, meet “Adam”, who told us “I worked for a regional internet service provider back in the mid 90s. The internet was just becoming a ‘big thing’ and we proudly boasted that we had about 25,000 active dial-up users at any given time.”

But under the hood, things were a bit rough.

“Our Network Operations Center was really just a small house in the middle of town with a lot of telco infrastructure feeding into a small 15x15 room with piles and piles of servers, switches, and a few big routers,” Adam told us. “All the equipment was stored in custom 'racks'. If you could call them that. They were some sort of metallic box with rails - but they weren't for rack-mount servers, they were for drawers. Almost like wide filing cabinets.”

Thankfully, the company eventually decided to upgrade into real racks. The plan was to “slide 2 by 4 slabs of wood under the bottom server, then slowly unbolt all the servers and lower them on to each other.”

“Then a staff member on either side of the rack would lift the 2x4s so the old rack could be cut away with a saw and the new rack could be slid into place. Then one-by-one the servers and switches and routers would be secured to the rack. Then we could let go of the 2 by 4s.”

Because Adam was young, fit and assumed to be strong, he scored the job at the back of the rack holding one end of the 2 by 4s.

“We managed to hold up about 120 pounds of servers for 15 minutes while the old rack was cut away and the new one was assembled around us,” Adam told Who, me?

“You can imagine our relief when the final piece of equipment was bolted into place and we could remove the 2 by 4s.”

Indeed, it was high fives all round because 25,000 active users were still online!

But as Adam crawled out from behind the rack he told us “My foot snagged something.”

“I froze and prayed, but God was not kind.”

“It was our main router. An old, clunky Cisco that took eight minutes to reboot.”

Customers noticed rather quickly.

“I had never seen 30 primary rate interface cards light up with phone calls at the same time before. It was like someone added an extra sun to the room,” Adam said.

“Fortunately my boss had been in IT longer than I had. He knew just how long it took to reboot a 486DX running Windows 95, so he raced out to the support call center and wrote in big letters ‘HAVE EVERYONE REBOOT AND TRY AGAIN. NOT AN OUTAGE’."

“By the time people had finished restarting, dialling up to our service, waiting for the modem negotiation tones to finish and then letting Trumpet Winsock negotiate, the router was back online.”

“Fun times,” Adam ended his email.

How have you fooled users into thinking your problem is their problem? Click here to write to Who, me? and we might just run your story here on a future Monday. ®