Reg reader escapes four-month lightning-struck Windows Vista farm nightmare
Know-it-all boss knows nothing
On-Call This week's instalment of On-Call, our regular reader-contributed tale of things that go pear-shaped in the small hours, comes from Carl who tells us that “a couple of years ago I had just been made redundant”.
Carl's the kind of chap who sees opportunity in adversity, and so felt it was his time to step up from sysadmin to IT manager, and landed a gig in the latter role at a company that designed and sold CCTV and time and attendance systems.
“The job involved supporting the company's IT and the internal version of their time and attendance software with the help of the developers,” Carl recalls. And it came with a rural setting: the office was in converted farm buildings adjacent to the owners home.
It looked like a good job going in. Carl says that in the interview he was told his predecessor had left “due to some family issues”. His employers did, however, mention that “there would be a lot of work to get the systems working smoothly as the IT manager they had before him had been sacked for incompetence.”
How incompetent? Plenty: this chap removed the RAID config on the email server to give them more storage space and, unsurprisingly, when another disk had failed the email server died. “On trying to restore the email server, he'd realised the backups had also been failing for month/years”.
Carl's first day rolled around and he quickly learned that his immediate predecessor had left the building and indeed had only been in it for two months. Which set him worrying.
Justifiably, as “despite being told that the previous IT manager had been there for most of the year, I found documents and invoices signed by two other IT managers, and they confirmed that in fact the company had been through three managers, not one, in the last 18 months.”
After a few days of intense reading about the IT infrastructure Carl “felt ready to jump in, only to find that a number of the credentials in the documentation was incorrect or incomplete, meaning a lot of the systems needed reconfiguring, without any way to check the current setup.”
“The day after I find this out I turned up in the morning to find that we'd been hit by lightning (the owner had been informed he needed a lightning rod, but had decided he knew better).”
Carl can now tell you that lightning does indeed strike twice in the same place. Or thrice in his case: the office copped two more jolts over the next month, “killing a few devices with it".
“Despite all of the above I continued to try and get the company IT to a decent (or at least not horrifying) state,” Carl said. Matters weren't helped by the fact that the company's owner was the managing director's father, using a PC running Windows 2000. In 2013.
When that venerable instrument died, Carl was accused of breaking it and told to fix it, again running Windows 2000.
“Doing this took eight hours, as they had not kept any Windows 2000 install media and none of the software he needed to program had been kept,” Carl recalls. “All I could do was replace DLLs in safe mode, reboot and see if it worked, if not, repeat.”
“A similar story can be told about the machine he used to control door access throughout his house, when one of the lightning strikes had blown up the motherboard and PCI card. I was informed there was a spare machine of the same spec, only to find out this had been used (and decommissioned) before I had arrived. Due to the card type, the only option for a replacement I could source was a Dell Optiplex GX270 for £600 (probably about that much when it was new) and then tried to help the developers reconfigure the software, which there were apparently unable to do.”
“After two days of the owner telling me we're losing tens of thousands of pounds in revenue (utter bollocks) and the developers using trial and error, we managed to get it connected correctly and working,” said Carl.
Other PCs around the business were in a state. Carl says most machines were five or more years old. The only “upgrade” for most was to Windows Vista. If 30-minute boot times for a Pentium 4 with 2GB of RAM can be considered an upgrade. Or even a PC.
When Carl pointed this out to management and requested funds for replacements "I was refused and told that the machines were fine". Carl persisted and convinced the managing director he should at least outline the costs of a new PC fleet, so prepared a proposal and was rejected. Utterly.
“The MD said that he didn't believe me ... I pushed for an explanation, and was told that he didn't believe I was lying to him, but believed that I was being lied to and didn't know what I was talking about. I pressed him further, at which point he got a bit irate and said 'They're all 3GHz processors and they would be no improvement on the current machines'. Thankfully, another manager in the meeting informed him he was wrong, but that was the last straw and I handed in my notice a couple of days later.
Carl said the above are “only the highlights of my four month stay". He's since been made permanent half-way through a contract and says he's “earning significantly more for doing a lot less work".
Sounds like the wheel of Karma's turned for Carl. If you've a similar story of workplace wonder, do get in touch. ®
Sponsored: Becoming a Pragmatic Security Leader