CEO insisted his email was on server that had been offline for years
When the dotcom bubble burst, the surviving techies learned the true meaning of just-in-time training
On-Call Welcome again to On-Call, The Register’s Friday column in which readers share tales of tricky tech support tasks.
This week meet “Sam” who “used to work for one of those zombie web companies that limped along after the dotcom bubble burst because no-one had told it that it should be dead.”
The company no longer had a proper sysadmin, so proper IT operations “had been replaced with frequent bursts of panic whenever something fell over.”
Which was quite often, because one of Sam’s colleagues “was one of those types who can pick up any language in seconds and create complex projects in next to no time.”
“The only problem was that he tended to write code with very little error checking while the whole company was basically in a permanent error state.”
The lack of sysadmin also meant that the company endured things like servers running out of space, which caused issues like a data loss near miss on a client project that Sam said “was basically a giant curated address book.”
“Every night it grabbed a .zip file and extracted XML files to create an updated list of contacts. If any contact had ceased to exist, the contact was deleted from everyone's address book. We even kept a copy of the .zip files for accountability purposes, even though there was no need to.”
Sysadmin’s worst client was … his mother! Until his sister called for helpREAD MORE
This worked until the company’s server ran out of disc space and the application started to unzip files that couldn’t be stored and then “swapped lots of full database tables with new empty ones, then went looking for invalid relationships.”
Which thanks to the missing data was all of the relationships. At which point the application deleted everything.
Backups? Remember that this company had no sysadmin. But it did have Sam and he got things back into a serviceable state after a week of heroic action.
You’d think the company would have learned from that.
Of course it didn’t: another of the address book’s functions was to watch for bounced emails to help keep the database clean of dead addresses.
A noble idea that. But “One night the mail server corrupted its disc and fell over. The task of bringing it back up landed on my lap and I started my journey from vaguely knowing how
grep worked to fixing broken mail servers.”
Sam did a lot of Googling, “while flinching every time the aircon wheezed, convinced I was about to be haloned to death.”
“Finally the mail server came up and I watched the first messages start rolling out. I went home.”
But the following morning “the client rang in a panic. They were getting a huge number of bounce message from us.”
Sam, now the “resident expert” was pressed into service.
“It quickly became obvious what was happening. When the code found a new email it sent on a bounce message and marked the email as read. It didn't delete it or move it to a new location - it just marked it as read. And it didn't check that it had actually worked. When the mail server crashed, it trashed the table that held the read flags, marking them all as unread and refusing to update them.”
Every few minutes a
cron task went looking for unread emails, found two years’ worth of them and tried to send them all on to the client. It then failed to mark them as read and repeated the task on the next
“I never found out how many emails we'd flooded the client with, but by the time I shut it down it had eighty seven thousand emails in its outbound queue and the
cron task was failing to finish before the next
crontask was triggered.”
'I crashed AOL for 19 hours and messed up global email for a week'READ MORE
This incident had a fun sequel a year later. By this time Sam’s company had moved to Gmail. He’d kindly “provided everyone with detailed instructions on how to transfer their email across.”
And with that done, he announced he would decommission the old mail server.
“But the CEO refused to let me take it offline, claiming ‘it contains old emails I need’.”
The last Sam heard, about four years after he quit, the CEO still insisted the mail server remain online.
“But he still hadn't noticed that I'd pulled the network cables before I quit. It was still on and he was happy.”
Have you worked in a more messed-up workplace than Sam’s dot-bomb? If so, write to On-Call and we’ll try to slip your story in here on a future Friday. ®
Sponsored: Becoming a Pragmatic Security Leader