Dead LAN's hand: IT staff 'locked out' of data center's core switch after the only bloke who could log into it dies
'We can replace it but we have no idea what the config is on the device'
An IT department is pulling its hair out this month after realizing a coworker who died last year was the only person who could log into a crucial network switch.
This is according to Dylan, a sysadmin at a small US healthcare company, who today told El Reg a story of how he and his colleagues ended up locked out of the equipment, thanks to an untimely death in 2018. The deceased, who we will refer to as Nick out of respect for the departed and family, was the only person who knew how to log into the core switch, and had a history of both health and work performance problems, it is claimed.
"Nick was absolutely not a good engineer," Dylan said. "He knew enough to get himself by on tasks such as setting up an incredibly insecure FTP server, creating an incredibly unstable ESXi environment, and pushing out ManageEngine products in insecure and often ridiculous ways."
By 2018, Dylan told us, Nick's haphazard network management had caused a bevy of problems. Already planning to replace the techie, managers were at the time struggling to fix an expired ESXi certificate Nick had neglected to replace when they received the news he had suddenly passed on.
Now, not only were Dylan and others tasked with figuring out how to manage, repair, and rebuild the unique network crafted by their late colleague, but they would have to do it without any of his login information or configuration details, according to our tipster.
"I started at this company just a month before Nick died so I got to experience this entire mess fold out and had to work with users on all the different things that were going down or breaking in the next few months afterwards," Dylan said. "Certificates expiring, lost private keys for VPNs, bad configs causing failures, and I worked hard at it. For my work, they promoted me to system administrator, as I was supporting a whole bunch of these strange applications anyway."
The secret switch
Things were looking up and going relatively smoothly until February this year, when Dylan noticed another thing wrong within the network at the company's remote data center.
"I was running a network scan in our data center and noticed we had a x.x.x.2, an x.x.x.3, and an x.x.x.4… What did the .4 go to? We had logins for .2 and .3…. But we weren’t sure what .4 was," Dylan told us. "We knew we had a Dell core switch in our data center so I asked around if anyone knew a login or had touched the thing. Our engineer gave me a login but said he hasn’t managed to get into it. This core switch was routing everything in our data center, and in the intervening year after our old engineer died, nobody has touched the thing."
Dylan said a technician was dispatched to the data center to sort it out, and they worked with Dell support to try "about 15 different" ways to get in, to no avail. A simple rip-and-replace of the switch is not possible either.
"We have a four-hour replacement on the device. So while they said they could get us one out there, the only problem is we have no idea what the config is on the device," Dylan said.
All good, leave it with you...? Chap is roped into tech support role for clueless customerREAD MORE
"No clue about VLANs, no clue about if it has STP, or trunking, or anything."
Barring a breakthrough in obtaining a valid login, Dylan told us the company has some scheduled downtime in April. Until then, they're simply going to hope the old Dell router holds up.
As far as advice for other admins, our source has a simple recommendation.
"For the love of God, backup your configs on your networking equipment. Back them up, backup your private keys for PuTTY so other people can login, buy a password manager and give your master key to your director of IT," Dylan offered. "That way if you die, someone can follow up on all of your work with all your passwords, private keys, and configs in case of a replacement situation."
Wise words, indeed. Good luck, Dylan. ®
Sponsored: Becoming a Pragmatic Security Leader