Why are there so many Windows Server 2003 stragglers?

Windows Server 2003 is almost out of support, and many of us simply don't have the option to upgrade to a newer operating system. In some cases this problem is self-imposed. In others it is the result of events beyond our control. Either way, there are millions of businesses – mostly small businesses – who simply don't have the option to upgrade even if they wanted to.

An inability to upgrade to a newer operations system is usually down to a business critical application not being ready for upgrade. In a perfect world, we'd all simply migrate to supported applications. This isn't always possible.

I've seen plenty of situations where industry specific software replacements simply don't exist. This usually happens in mature markets that have reached economic entropy.

Markets tend to go one of two ways. Where delivery of a good or service is difficult - such as delivering utilities across a geographic area the size of a Canadian province - natural monopolies form. Small companies merge, are acquired and so forth until only one or two companies dominate. These guys have no problems writing new software to replace the old.

The other option is economic entropy. No individual company ever rises to the top. An unlimited number of small, local companies compete viciously with one another, driving down margins to the point of ridiculousness.

This latter economic option is reasonably common in Canada. There are a number of markets where no one company has enough money to take have their 30 year old industry specific software rewritten.

There is no money in a software house rewriting the software on spec and then trying to sell it into that market because none of the companies in that market have any money. There's no hope of getting all the companies in the market to work together towards getting new software written because they all hate each other.

Server 2003 going "end of life" is an ephemeral concern. It ranks so far below "how are we going to make enough money to make payroll for the next 6 months" that it is considered a waste of time to even discuss the issue.

Legal considerations

Many other situations exist where companies simply can't afford to upgrade. Upgrading the operating system means upgrading the application software which means replacing expensive (and often physically quite large) equipment that can run into the millions of dollars: fine for a multinational conglomerate, corporate suicide for many a small business.

If the developer isn't ready for the new operating system (or no longer exists) then several factors have to be considered. The most pressing issues preventing upgrade usually aren't technical, they're regulatory and political.

If your company requires fully vendor-supported applications in order to meet regulatory requirements then a vendor who isn't ready to move off of Server 2003 is a huge problem. That developer could be putting your organisation in a compromising – and potentially expensive – position.

There may also be resource or political issues within the IT department. Few administrators want to be responsible for an application that the vendor no longer supports. It is very time consuming and far more personally risky to the responsible individuals should something go wrong. Assurances may have to be made and indemnifications granted if a company wants to drag along software that's out of support.

Secure or die

Outside the regulated industries "adapt or die" is the mantra. Small business sysadmins are frequently told to make do with what they have...and many of us have done so for decades.

I recently inherited a site with 6 Windows NT servers and a Novell Netware setup that functionally can't be upgraded. I have several other sites where Server 2000 is still in play and it works just fine, thank you very much. There's nothing wrong with these systems and I am quite convinced they are just as secure as their newer brethren, but only because precautions have been taken to make them so.

In almost all cases where something like Windows NT, Server 2000, Server 2003 or Netware have to be dragged along it is because there is some very specific application or piece of hardware that they have to run. That's fine, but to be perfectly blunt about this: there is no way we can trust those operating systems at any point to not be compromised. To continue to work with them, we need to treat them as potentially hostile and isolate them.

I rely a lot on virtualisation and ghost. Keep the operating system and application installs separate from the data and configuration as much as possible. Regularly back up the data and configuration files. Periodically reimage the systems back to "known clean" and reinject the data and configuration files. Where possible, I do this nightly.

Isolate the systems physically. Disable USB, optical and floppy drives and any other ways your employees might have of "accidentally" introducing malware onto the systems.

If the systems need network access, make it highly restricted and on a completely separate and heavily defended network. Firewalls and intrusion detection systems should be segregating those systems from the rest of the network.

The rule of thumb is to minimise the number of possible ways those systems can interact with the outside world. I use MAC address filtering to limit the systems my insecure OSes can talk to. For the handful that need internet access (usually some system running a manufacturing machine that needs to call home to report how many widgets have been created) I forces them through a highly secured proxy system and they can only access whitelisted IP addresses and domains.

If possible, build a highly secured gateway system. Typically a terminal server. Your employees interact with the gateway system (such as logging in to the terminal server) from one physical network; in turn, the gateway system accesses the insecure systems through a second physical network.

Horses for courses

I'm not sure I'd be comfortable with my bank secretly running a bunch of Windows NT boxes. At least not if they were physically in the same building as the systems which held my banking data, or had access to the internet. I'm pretty sure I'm uncomfortable with the idea that my medical records might be held on Server 2003 systems about to hit the end of life as well.

Banks make rather a lot of money. Health care systems in non-dystopian nations are government funded. There's no excuse for these organisations to be behind. Software to scan networks, or even hiring an intern to physically check each box in the building isn't really all that expensive.

On the other hand, I don't care at all what my bakery runs. If you're really terribly interested in where I buy my food you're either running an academic study of some sort or there's something really, really wrong with you.

The biggest threat my bakery has to my personal info is the association of my phone number with my name (that's public record) and that when I pay for things I use a debit machine provided to them by the local bank. That information never goes into their computers; my bakery wouldn't know how to make that happen even if the bank were to allow it. (The bank would not allow it.)

If I get pwned at the bakery, it's going to be because someone put a skimmer in the debit machine, not because their ancient Windows NT-based computerized oven got an interesting piece of malware. Besides, at some point, that stuff is getting old enough that it's coming out the other side: so old nobody is developing new malware for it!

How upset we need to get about systems not being upgraded depends on what those systems are being used for and how they are being cared for. It's a popular mindset to believe that small business systems administrators are incompetent or won't defend their estate, but in my experience they are often passionate and capable, if spectacularly under-resourced.

There are some who are to blame

Pointed questions need to be asked about why well-resourced companies and government agencies are dragging along dated operating systems and/or applications that are no longer supported by their vendors. Anyone operating in a regulated industry can't claim poverty: every other player in the industry must abide by the same rules, so keeping IT up to date it is simply a cost of doing business.

In regulated industries IT costs are simply passed along to the customer. Everyone has to pay them and nobody gets to undercut competition by doing without. There isn't the margin-evaporating cutthroat nature of entropy markets at play.

Thus it's hard to find any excuses fronted by banks, health care systems and so forth acceptable. There is no way they did not know the end of life was coming. There is no way they didn't have the resources to cope with the change. In many cases regulated industries can't simply defend their unsupported estate, call it "good enough" and pass their various audits and certifications.

For those who don't have a choice about upgrading, the typical internet commenter approach of guilt, shame and fear isn't going to change the reality of their situation. Telling these companies and their systems administrators that they if they do not update they are morally, ethically and professional bankrupt is not helping anyone.

Our focus needs to be on helping these organisations with mitigation of risk, detection of breaches and having well-rehearsed incident response plans. In fact, we should all be doing this for all of our systems, but most of us won't until faced with having to cope with a clear and present risk.

For those who do have a choice about upgrading, we need to ask what plans are in place to perform those upgrades and what the timelines look like. More importantly, what plans exist to harden defences and mitigate risk in the time between when Server 2003 goes out of support and when they plan to have their act together?

Most importantly, we need to start talking now about the end of life of Server 2008, Server 2008 R2 and Windows 7. 2020 isn't that far away, and all these old problems will be new again. How can we help those for whom migration seems financially impossible? How can we help those who have the resources but based on current performance not the skill to organise and execute upgrades in time?

Maybe, if we work at it, we can learn from today's mistakes in time to solve tomorrow's problems. ®