Patching is a pain...
...but misconfiguration is worse
Sysadmin Blog After a couple of pretty bad weeks, in which virtually everything that could conceivably have gone wrong has, things are finally starting to settle down.
Despite a couple of “weeks from hell” in which my network survived virtually every “network down” scenario back to back, none of that actually bothers me. Some of these worst case scenarios were covered off by a bit of planning; others weren’t. Of those that weren’t, the vast majority of these problems could have been avoided if the network upgrades and maintenance currently on the drawing board had been completed. (Completion date for most of it is the end of July, go figure, eh?)
What bothers me are the little things that I have discovered wrong with my network these past few weeks. As my previous article revealed, last week I discovered a virus on my network. It was largely contained by my edge defences, and given all that decided to erupt at the same time, it was really little more than a minor inconvenience. Still, it irks me. Minor misconfigurations can have a dramatic effect on your network by allowing attackers or malware to gain a foothold.
The first issue I would like to address is that of patch management. Not the overall operating system patch management, but patching your individual applications. Most operating systems have an update mechanism built in. Microsoft Update will patch Windows as well as some other Microsoft products, but let’s be honest - compared to either of the two major Linux package managers it can only be described as unbelievably primitive.
Sadly, despite all the facilities built into the iStore that have simplified application updates on iPhoneOS devices, full-fat OSX users suffer the same level of third-party neglect that Windows users do. Even on Linux; if you install an application without registering it with the system package manager, you regular updates will not update that app.
Where this leaves us then is sadly right were we were a decade ago; the operating system will patch itself, some applications (such as Firefox) will do so, but an unfortunate amount of software you can run on your desktop will require you to individually hunt updates. For those applications that do update themselves, many do so via their own background update mechanism. These updaters can all be sitting in the background consuming resources and waiting for the next patch.
The end result of this approach to patching is that it is virtually impossible to centralise, yet because of the security ramifications of running unpatched software, (a certain PDF reader springs immediately to mind,) patching third party applications is absolutely critical to proper network security. Users won’t update applications; they become very quickly annoyed at the whole process, and who can blame them? Whether it be Adobe updater informing you of a patch, Java bleating that yet another version has arrived to break your application compatibility or Google delivering yet another refinement to its user tracking toolbar, there are quite simply always updates to be run.
For this, I wish I had some secret sauce to solve the problem. You can’t make users update, and neither Apple nor Microsoft have been particularly keen on making the lives of their users actually easy by opening their patch management systems to third parties. There are some systems management applications out there that claim to be able to help - but the list of applications they support never seems to cover what you actually have deployed.
If you are lucky enough to be able to run only applications with their own third-party patching tools, then take the time to configure them. Several can be set up to automatically patch themselves, and the better designed ones can be configured to report errors in patching to an email address. They aren’t all so helpful, and I am going to take a moment to pick on Adobe. I recognise that the chances my gripes about Adobe being read by anyone there are slim, but they really deserve a right good kicking, and frankly I need the catharsis.
Adobe brings to the table everything that is wrong in an updater. It doesn’t have a centralised management function, it doesn’t have an auto-install option, it doesn’t even have the ability to email if things go pear-shaped. What it does do is consume a lot of resources, slow application launches, pop up right in the middle of reading a PDF, take absolute ages to do patches of any kind and it looks awful to boot.
The only thing it does right at all is this: all Adobe applications can be patched in from this one updater. In all other respects it is an abomination; it actually just asked me to restart my computer in order to update a PDF reader. I am absolutely floored; I wonder how many millions of dollars of systems administrators’ salary have been wasted in how many hundreds of thousands of companies babying updaters like this? Adobe (and all other companies with crap or worse yet no updaters) - get your house in order.
Things aren’t all bleak, though. Two of my favourite programs (Firefox and Notepad++) have what I consider to be excellent updaters. When I launch the program, the updater pops up quickly (without noticeably degrading application launch) asks me if I would like to update, does so, and then relaunches the application for me. Even the slowest updates don’t seem to take very long. Because it is integrated into the launching of the application (rather than Yet Another Tray Icon that users ignore), I find that these applications do in fact get updated on a regular basis.
It does then become an issue of IT procedures to manage individual application patching. Some organisations have adopted “Patch Tuesday” as the night of patch testing and deployment, not only for Microsoft’s software, but to run around all desktops and patch them. For larger installations this isn’t practical; but mass imaging your desktops is. It might sound insane at first, but third party security holes are in many cases such an issue that patching a reference image and then pushing it out to your desktops once a month should be seriously considered as a bit of preventative IT medicine.
There are other common configuration mistakes that can lead to malware issues on your network beyond application patch management. The one I feel is the most critical to reinforce is changing the default settings. The most obvious example for a home user is to change the default settings of their wireless or internet routers. Ever time I see a wireless SSID that says “default” or “Linksys” a part of me dies inside. Worse yet, most of them still use default administrative passwords. Though some default settings prevent a wireless client from accessing the administration panel, others don’t.
It is entirely possible to flash the router in question with something like dd-wrt, change the settings to whatever the defaults were on the original firmware, and plant a rootkit on the router itself. A few hours of wardriving in a major metro and you could have yourself a kilonode botnet. Worse yet, users could try to scan all they want; they’d never find it because it isn’t on their PCs.
Leave firewalls up, even on protected networks. Open services required (such as file sharing) only to your internal subnet, but otherwise leave your shields up. You can’t trust that the other computers on your network aren’t infected by malware any more than you can trust the internet not to infect you. Even if you ran 15 anti-malware scanners on every system on your network (and it consisted entirely of Linux boxes) you could still only say that there was no malware present that you could detect. Better safe than sorry, and firewalls on more than just the edge devices are no longer optional in today’s computing environment.
For those that haven’t already, disable autoplay in Windows; this alone can save you quite a bit of grief. It is absolutely unreal how much trouble autoplay can cause on a Windows system where the user is logged in with administrative privileges.
If your log systems have the ability to email you on error, enable it. If they don’t, then get an application that can do so and install it. Analysing system logs can be the most important part of a systems administrator’s job. Security breaches leave traces, like fingerprints, if you know what to look for you can find out who or what broke in. Even if you don’t know exactly what to look for, when your logs start doing something odd then you know it’s time to do some research or ask for help.
Look into OpenDNS, or if you host DNS of your own look into malwaredomains.com. Incorporating DNS blocking either at the DNS level or the firewall level is an absolutely required element of minimal network caution. If all your other protective measures fail and you do get infected, then there is a really good chance the malware won’t be able to contact its command and control server, effectively neutering it.
Default ports for applications such as SSH, or remote administration applications are a must. It might only slow down a determined attacker, but if you have halfway decent Intrusion Detection System (IDS), they won’t get very far along in port scanning the system looking for holes before they are banned.
To round out your defences, look into network vulnerability scanners like Tenable’s Nessus or GFI’s Languard. (Both offer versions free for home use.) These applications can scan your systems looking for security vulnerabilities or malware activity signatures. Take the time to get to learn a few different vulnerability scanners, and most importantly, actually use them on a regular basis.
This leads me into software that can help you detect when your security is breached in near real-time. IDSes are offered by every internet security firm out there, as well as most major OEMs. Some are installable applications geared towards specific markets; home users, protecting individual servers or running on network edge devices. Others are appliances designed either to sit on the edge of your network or to monitor it silently from the inside.
Regardless of whose IDS you choose, get one. Get several in fact - back up your primary IDS with open source virtual appliances running iptables, snort and squid. Like anti-malware (AM) scanners, no IDS is perfect. Eventually an attacker will figure out how to poke a hole through one, or a piece of malware will have a signature the IDS doesn’t recognise. Multiple layers offer the best defence.
So to summarise the above; it is critical to configure your systems to actually inform you of breaches. RAID card management applications can email when a disk is dropped, even my UPSes email me when the power goes out. Network security breaches require the same sort of scrutiny. From patch management applications to error logs and logfile analysis programs to intrusion detection systems, the greatest tool available to solve network security issues is the systems administrator responsible for that network. In order to work their magic, they need to know first that a problem exists.
Sadly it is perhaps one of the most overlooked features in just about any application; either the developer neglected to include the capability, or the administrator doing system configuration neglected to enable it. Even if the system in your care is Aunt Tilly’s home PC, set it up to email you when there’s a problem - it might help you catch a breach before it really gets out of hand. ®