Sysadmins: Step away from the Big Mac. No more Heartbleed-style 2am patch dashes
6 steps to a saner patching regime
Give us the tools...
3. Get the right tools: manual patching doesn’t work anymore. Scaling above a handful of users is when patch management becomes time-consuming and error prone if not done in a managed way. Manually repeating patching on machines allows room for error. There are many patch management products on the market for administrators to use, and some are even free. The cost of any tool you buy will quickly be offset by the time saved over manual patch methods. The one unfortunate part of getting the right tools is that if you run more than one type of OS, you may need to get a second set of patching tools. For example, if you maintain a sizeable Red Hat installation you will need to use Satellite or something similar to automate the process.
4. Some machines are special and hand deployment has a place. Some machines are very critical and patching should be supervised or done manually. Applying the same patches should happen, just not automatically. To give a quick example, in a virtualised environment, there can be many dependencies such as AD, SSO, and similar. If you lose the ability to manage the VM estate from a single piece of glass you will need to start trying to figure out where the affected machine is. Therefore, apply the patches and ensure the machine is clean after a reboot. Only experience of your environment will tell you which machines may require enhanced management.
5. Test, test and test your patches before deploying. It may sound obvious, but fixing a bad patch can be very problematic and expensive, especially if deskside intervention is required. An ideal testing set-up should include three distinct groups. Essentially, a patch alpha release to those knowledgeable IT staff who can cope better with a bad patch. The second group should comprise selected machines and users that encompass users of each of the different builds used within the environment. The last group should be akin to the second group but with larger numbers, or indeed sites. It depends on the set-up being managed.
They may seem like time-wasting, useless beasts, but users are people too...
6. The users are people too. Users realise that bugs have to be patched, but it helps if you get them on board early. They would much rather be told a week before about the patch so they don’t make plans to work at the weekend and they suddenly can't, as the network is down. Also, plan these patches at appropriate times. Month end is not usually a good patching time for accountants, for example.
The fly in the ointment: appliances
Working in larger environments and ensuring patch compliance can be very interesting when applied to appliances, both physical and virtual. All major operating systems have some form of patch management rolled into their management tools. However, appliance manufacturers who run CentOS or some other Linux variant will often only support their own patch update channels and provide limited support for other management tools, crying “unsupported configuration” when you broach the subject.
This may not be an issue if you have only or two devices, but what if you have several dozen appliances, with one appliance per virtual host to provide a specific service, for example? It gets tiresome and boring, but can be essential. Manually logging into each device several times in order to update it gets really boring, really quickly. Then what if you have tens of appliances from several vendors? More time wasted. Applying patches can quickly become a major logistical nightmare when done globally. At present there are not many tools that will patch appliances from several vendors AND are vendor supported to boot.
To be sure, things are improving with centralised web GUIs to make updating appliances easier, but we are not there yet. Given time and the number of recent Linux security bugs, I think vendors will need to move towards a more integrated patch management model, above and beyond what a lot currently provide.
Patching is one of those activities that is essential to any environment of any size. It is without doubt a cost to companies that provides no extra business edge, but can’t be avoided. Using tools such as SCCM or Redhat Satellite can help manage the environment and reduce deployment overhead.
That represents only half the story, as the management overhead and business requirements can easily push the cost of a patch up significantly. But it is a cost that has to be shouldered.
Optimising it to be as efficient as possible is worth the investment in time, because it is a certainty that these steps will be repeated frequently, month to month and beyond. ®