It's a pain in the ASCII, so what can be done to make patching easier?

Like most of you reading this article, I neglect good patching hygiene.

There are very good reasons why we should all of us obsessively test every patch and patch our systems immediately, but patching is a pain in the ASCII.

The tools suck, rebooting sucks, and most damning of all, something usually breaks.

Each PC, and most servers, have more applications installed today than they did back when I were a lad. Everything is also far more interconnected and interdependent. In today's world, patches cascade, and there are ever so many of them.

The corner store stopped selling the 1.0 lightbulbs you were buying and only sells version 2.0. Lightbulb 2.0 needs a new client application to talk to it. Client applications must be in lock-step not only with server applications but with rapidly evolving web-based APIs as well.

Multiple authentication and identity sources are baked into virtually everything – each a separate API, client application or server of its own to worry about.

After rebooting what seems like half the computers on the continent you finally get everything updated

Server applications are often a database, a web server of some variety and a set of binaries running on one or more operating systems spread across who knows how many hosts.

Each host has its own hypervisor. The hypervisor has its own management server, which has its own client application which hooks into those authentication sources. After rebooting what seems like half the computers on the continent you finally get everything updated to where it is supposed to be.

Only to find out, of course, that the new client application isn't backwards compatible with your installed base of 1.0 lightbulbs. Patching is still something of a shambles.

Stop the world

Rebooting is a huge part of the problem with patching. All operating systems have the issue to some degree, but Microsoft's Windows causes the vast bulk of the frustration.

When you are patching a server you require one of two things. You can have an expensive high-availability infrastructure that ensures you can take down individual copies of whatever server application you are using without a loss in service; or you can schedule maintenance windows.

The infamous Patch Tuesday is something all of us are all too familiar with. Maintenance windows are few and far between. Ubiquitous broadband and usable mobile data capabilities have seen an increase in people working remotely.

Companies, employees and customers expect 24/7 access to everything. For many of us, scheduling downtime takes almost as much time as the patching itself.

It is even worse on the end-user side. Rebooting desktops is a right royal pain. Looking at my own desktop, it has been pending updates for about four months now, even though I know full well how important patching is.

If I wanted to patch this system I'd have to organise and save all the scraps of data in the dozen or so open notepad instances, then remember to re-open them when I log back on. I'd have to deal with the seven emails that seem to be perpetually open.

I'd have to save, organise and reopen the articles I have open in Writer, dig up that code so I can reconnect PowerShell to Office 365, reopen Windows Explorer to the four folders I keep open and jot down any of the necessary information from the "off the record" IMs and chats I have running.

In short, it's a hassle (obligatory XKCD). I just want my computer to work and I don't want to take the better part of an hour saving, sorting, closing, figuring out what needs to be patched, patching, rebooting and then opening everything again.

As a sysadmin I really should kick my own ass for displaying such an attitude. I certainly wouldn't accept that from any of my users. There has to be something we can do to break the deadlock.

Too many holes

None of us can do much about the need for Windows to reboot. Microsoft's response to the issue is to invent any number of technologies to make the startup process faster, while ignoring the real sources of frustration that prevent users from rebooting.

This dance will continue for some time, but it is not a battle that we, the customer, are going to win.

Software developers can make a difference. Proper web browsers like Chrome and Firefox now remember the tabs you had open upon reboot and will "restore previous session" when the system is back up.

If we can encourage other developers to take a similar approach we might well get users updating more frequently.

One reason why I don't patch all that often is simply because it takes forever to find all the things that need to be patched. Windows might want a patch, but when I reboot a half dozen other applications are going to run their "on boot" update systems.

This will remind me that I have 30 or so applications on my system that don't have updaters of their own, and now I have to go see if they are out of date.

One critical thing we can do is to get proper patch management tools. Patching and rebooting is so frustrating that anything we can do to minimise the number of instances it must occur is a decidedly good thing.

When I talk of patch management tools, I am not talking about Microsoft's Windows Server Update Services. It is cute, but it covers only Microsoft products. Many of the applications on our computers don't have update programs of their own.

Those that do range from mind-bogglingly irritating to barely functional. A single application that handles patching for the operating system as well as third-party applications is required.

Testing times

Far too many of us simply don't test patches. In some cases this is due to limited lab space, but often doing the update dance takes so long that it is worth taking a risk.

Rolling back changes because of a patch issue is something we have to do only every year or two. Patch testing can take days out of every month.

One of the most important benefits of proper patch management applications is one that only systems administrators will see. If we can find all the applications on a given system in one go then the burden of actually testing these patches in the lab before releasing them is lowered considerably.

Similarly, if we can push patches to our users – or ourselves – as a single lump once a month and know we are up to date, then we can get into the habit of setting aside some time each month to make it happen.

None of us is going to like patching, but if we get the proper tools involved, at least we can make it bearable. ®