The self-disconnecting switch: Ghost in the machine or just a desire to save some cash?
Yet another reason to never do things by halves
On Call The weekend is a day away, but before you swan off, please join us for another episode of ticketing system terror with The Register's regular On Call.
"Forgive the beancounters, for they know not what they do" might chime with today's contributor, Reg reader "Arnold", who found himself (and his customers) on the sharp, pointy end of the some ill-considered cost savings.
Arnold spent the early part of this century working for a now-defunct ISP in continental Europe. He had been employed as a network designer, but "also had to step in when difficult or large problems arose in the network."
While the network was sensibly spread over several data centres, much of the internet peering happened in a single, specific location. Not in the commercial data centre, oh no, but in a building across the street, supervised by an "academic authority."
You'll never select all and mark as read again after this tale of peril... Oh, who are we kidding? Of course you willREAD MORE
"To avoid deploying new fibres to the other building for each new peering agreement we signed, we had one big monster Ethernet switch in the actual peering room in the academic building."
The switch was connected with two fibre pairs, with separate geographic routing to the commercial data centre. It had redundancy up the wazoo, and there was even a UPS to augment things.
"What," asked Arnold, "could go wrong?"
Eagle-eyed readers will have noted the word "one" in an earlier paragraph, which will give a clue as to what happened next.
The On Call moment came knocking at a very civilised 10am on that fateful day: "The big Ethernet Switch suddenly completely disappeared from our management system."
All peering was lost at the location. Sure, there were backups elsewhere, but nowhere near enough to cater for all the ISP's customers. Buh-bye connectivity.
The management tools showed Arnold that there was no signal on either fibre from the peering location. An engineer was dispatched, but would take a long half-hour to get there.
"So I called the data centre engineer on duty to ask him if they were doing some work on their fibre patch panels."
Nope. There was also nothing coming from the building opposite, where the peering room was located. Nor was there a surprise digger in the street, accidentally ripping up precious cables. Doubtless aware that customer anguish was increasing exponentially, the data centre engineer volunteered to pop across the road and see if there was a problem there.
Five minutes later he called back. Not only was the Ethernet switch off, it had somehow also managed to launch itself from the rack and was now sat on the floor with all fibre and power cables disconnected.
Plugging the thing back in (which took a good half-hour of fibre fiddling) brought service back to anxious customers. However, the question remained: how had this happened?
The reason could be traced all the way back to deciding that a single Ethernet switch would do the job. Somebody (probably with a head for figures) had piped up with: "Why rent a complete rack if you can also rent half a rack, which had ample room for said switch?"
And so it was that the upper part of the rack was rented to another company, and their equipment installed. Everybody was happy, right up until (unbeknownst to Arnold & co) that company went bust.
"The liquidators of that company, trying to recover some money for the creditors, noticed the assets present in the peering room," recalled Arnold. However, they neglected to tell the engineer instructed to seize the gear that only half the rack was theirs.
And so it was that customers found themselves rudely ripped from the internet as the ISP's Ethernet switch was torn out, along with everything else from that rack. They could only count themselves lucky that it was caught before popping up in an auction.
Still, the beancounters had the last laugh: "To prevent such a thing from happening again, we could rent the complete rack in the peering room for the same price as we paid for the half rack from then on."
Ever come up with a wicked wheeze to save some cash with some creativity in the racks? Or been undone by the bailiffs? We couldn't possibly comment, but you can. Send an email to On Call with your tale. ®