Feeds

Confessions of a sysadmin

I found a virus on my network today…

  • alert
  • submit to reddit

Secure remote control for conventional and virtual desktops

Blog I would like to say that it has been a few days since my last malware infected computer. I have been dealing with a string of these lately, and I’ve had quite enough of them for now, thank you.

I would also like to say my network was the epitome of configuration perfection, with every system fully patched, and a team of network ninjas facing off against hired pirates in a never ending battle for security perfection. The truth, however, is less ideal. My network has some systems that can’t ever be patched, and others where IT can’t force automatic patches. Configuration errors will inevitably exist due to a combination of lack of time, lack of knowledge or prioritization of IT tasks.

According to the email of article topics in my inbox, this one is supposed to be about the importance of proper configuration and patch management. Instead of being able to stand atop an ivory tower and reveal to you the secrets of perfect network management, I am forced to humble myself before the entire internet with a confession:

I discovered the Conficker worm on my network today.

I am shamed by this because the infection was entirely preventable and all the more because this discovery occurred the day this very article was due. While I had a lovely sermon prepared in which I would discuss why proper configuration and patch management are so very important, I think that doing a post-mortem on exactly how I contracted these bugs will be both far more entertaining, and perhaps even a little enlightening.

I discovered the infection today on Windows 2000 systems running Service Pack 4. Each system serves as a network and command and control interface for a large piece of equipment (think the size of a small car).

The hardware they are running on is fairly old (they talk to their attached equipment via a truly ancient SCSI card) and the software is remarkably picky and brittle. If installed exactly as directed, the computers (and their attached equipment) run just fine. Install the wrong windows update or change the wrong setting and they will refuse to work.

As an added bonus, the hardware specifications on the provided kit is so exact that if you were to (for example) load an anti-malware scanner on the system then the performance decrease would very negatively affect the productivity of the unit. Any decrease in output capacity of these units simply will not be tolerated.

The more I delve into the situation the more I am convinced these systems were infected a while ago. We had a user who opened an infected attachment (Windows XP, and yes they had to be running as an administrator to get their work done). For the curious it was a pdf. This turned out to be Conficker, which in turn ate every vulnerable computer on the entire network in about 15 minutes flat, and a fun night was had by all.

After we had sent the initial conficker infection shrieking back into the void from which it arose, we ran around to every single computer on the network and checked them one at a time. We remoted into each one in turn, ticked them off against both our IT internal list of systems, DHCP and even a Languard scan. After a few hours of fighting this particular brushfire, we were satisfied the network was clean and went home. By the time we arrived the next day we were on to the next problem, and the infection was almost completely forgotten.

This is where I made a big mistake.

The systems I discovered as infected today were, at the time we started cleaning the network, simply turned off. At the end of every work day, when the staff who use that equipment are done with it, they shut it down. They must have been active when the initial infection took place, and were turned off by staff member leaving for the night after we booted everyone off the network.

What’s worse, I completely forgot that those systems had Windows computers in them. They were, as computers integrated into larger pieces of equipment, out of sight and thus out of mind. (Let that be a lesson to you all: computers are integrated into everything these days. Think really, really hard about what’s on your network before declaring it bug-free.)

Knowing how these systems got infected, let’s delve into how I could have prevented this from occurring. The first and most obvious problem is that of patch management. I have a Windows Server Update Services (WSUS) server on my network to distribute patches, and I am very fastidious about testing patches against existing software and releasing the updates as soon as possible.

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
Stop the IoT revolution! We need to figure out packet sizes first
Researchers test 802.15.4 and find we know nuh-think! about large scale sensor network ops
Turnbull should spare us all airline-magazine-grade cloud hype
Box-hugger is not a dirty word, Minister. Box-huggers make the cloud WORK
SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016
Flash WORM has a serious use for archived photos and videos
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
Microsoft adds video offering to Office 365. Oh NOES, you'll need Adobe Flash
Lovely presentations... but not on your Flash-hating mobe
prev story

Whitepapers

Designing and building an open ITOA architecture
Learn about a new IT data taxonomy defined by the four data sources of IT visibility: wire, machine, agent, and synthetic data sets.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
5 critical considerations for enterprise cloud backup
Key considerations when evaluating cloud backup solutions to ensure adequate protection security and availability of enterprise data.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Driving business with continuous operational intelligence
Introducing an innovative approach offered by ExtraHop for producing continuous operational intelligence.