Switch survives three hours of beer spray, fails after twelve
Reader recalls a series of own goals doing tech support for a big football club
On-Call If it's Friday it must be time for On-Call, our regular reader-written romp through fun times at work.
This week, reader “Oswald” shared the story of the time he worked “for the IT company that looked after a well known football club.”
Oswald's a Brit, so when he says “football” he means soccer.
The club was his employer's trophy client, mentioned in every piece of marketing material and over-serviced lavishly and expensively with extras like staff being despatched on match day to make sure there was not a moment of downtime.
Oswald was “second engineer” on the account, which meant he could be sent out to do nasty things by his boss. Like the time he was asked to visit a player's house to figure out why the broadband was down. It turned out the player, who spoke little English, hadn't paid his bills.
Football clubs employ fixers to sort out that kind of thing, so Oswald deftly passed that problem to the “player assistance manager”.
Oswald was asked to visit the player by his boss, who we'll call “Ted”. Oswald calls him a relic who “had not adopted any new skills since NT4 as far as I could tell and had never managed anything larger than a site with 30 users before.”
This made for some ticklish times, such as the occasion when Ted was confronted with a problem: the food and booze kiosks at the club's ground did not have enough network ports to handle all the cash registers and credit card readers. Ted's answer was to put in some eight-port power-over-ethernet switches.
Which he put next to the beer taps.
It turns out that a POE switch can handle a football match of beer spillage. But Oswald was left to discover that they can't handle the twelve-hour drenching that comes with two twelve-hour days of a music festival. By the end of said event's first day, all the spare switches were already in use. Several outlets had to revert to cash sales only.
Ted also decided that the club's Exchange Server didn't need to monitor how much disk space was available. Which went predictably well once free space fell to under 10GB.
Oswald got the task of ordering a lot of disks, fast, because football clubs tend to have critical requirements when things like transfer deadlines come along. Along the way he checked out the configuration and found 500 mailboxes all sharing a single database. Had that arrangement persisted, Oswald thinks data loss was inevitable. Thankfully, he was able to save that one off the line.
Oswald's moved on, but says “To the best of my knowledge Ted still works there and they still think he does a good job.”
“They still have the club as a customer and I can only assume they are still getting bent over at contract negotiations. I left for a better role else where and regularly use tales of Ted to warn my team how bad things can be.”
How bad have things been with your team or clients? Write to me and we'll try to slot you into the back of the net in a future On-Call. ®
Sponsored: Becoming a Pragmatic Security Leader