Sysadmin ignores 25 THOUSAND patches, among other sins
'If I ever see this bloke on a job I'll punch him in the face' says Reg reader
On-call And that's one of the easier chores our reader found himself faced with in a new temp job. Most weekends, our On-Call feature looks at the odd situations readers find themselves in when called to do something on a client site or in the dead of night.
This week we're making an exception for reader “Bill”, who rates himself as “just your average support engineer” with experience on the front lines at companies big and small.
“I have learned the hard way about things over the years as we all do,” Bill says. But nothing prepared him for a recent gig he describes as “support for some very, very important persons in a small office that accounted for about 40 per cent of the money flowing out of a large multinational”.
The job looked simple: for a month, he'd be supporting Windows Server 2003 and 2008 for 50-60 users with onsite Exchange, Active Directory and BlackBerry Enterprise Server. At the end of the month, a new full-timer would be aboard and Bill could move on to his next contract.
Day one was encouraging. After “all the spiel and introductions companies normally muster for more important people on their first day”, plenty of it in the boardroom, Bill was “handed an A5 piece of paper with the layout of the servers on site and some IPs and hostnames”.
Which is where the trouble started, because that document was maybe 40 or 50 per cent accurate.
Once inside, Bill found things in quite a state. A WSUS server he felt should really have its own box was running on the company's main SQL server that also hosted core financial applications. The WSUS box also had 25,000 patches – Bill's certain it was thousands, not hundreds – awaiting approval or declination.
Getting that sorted was his first day's work, at the end of which he noticed another small SQL box with historical stuff on it, which was running Windows Server 2000. The physical server was made in 2001 and boasted a built-in floppy drive.
Will left the WSUS server “buzzing the 5 Mbps WAN link overnight” and arrived to find "users not happy that they had to restart their machines, and some had a few hundred updates to apply”. Some were even running Windows XP and had also been left unpatched.
For about four years.
Next? Check the anti-virus. It was 15 months out of date because “at some point a firewall rule for it was disabled and now couldn't be re-enabled”. Bill's predecessor hadn't left the firewall's password behind, starting with a need to swamp the connection daily to access new virus signatures.