Feeds

Belt and braces stop the network falling down

Count everything, then double it

Choosing a cloud hosting partner with confidence

Lately, everyone seems to have lined up to join the network failure party. In some cases, lax network security has been to blame.

In others, upgrade issues coupled with fundamental design flaws have done the damage.

An inability to cope with denial-of-service attacks by angry internet mobs has even resulted in disruptions to networks that are arguably among the most important on the planet.

The list goes on and on, and it would all be an amusing farce if the results were sometimes not so serious.

The failure of so many major corporations and government agencies to prevent network mishaps is far from an excuse for regular sysadmins to slack. Quite the opposite: these events highlight the general public's increasing unwillingness to forgive such breakdowns.

Whale of a time

Network outages lead to damning press coverage that can tarnish a brand for years, and even lead some to question the maturity of cloud computing. Network failure is so common that it even has its own whale mascot.

At a minimum, network redundancy requires two of everything: every switch, router, network card and cable on the network. Ideally, there should be three of everything. This allows you to take one set of equipment offline for scheduled maintenance with both a primary and backup remaining active.

Preventative maintenance touches on an element of network redundancy far more important than the choice of hardware. The single most important element of network redundancy is a philosophy.

We in IT don't get to live by the axiom "if it ain't broke don't fix it". It is always broken. Even when it doesn’t appear to be broken, there is bound to be a security flaw in the code somewhere that you will eventually have to patch.

Go forth and multiply

This means that while there should be three of everything, there really should also be four of almost everything. The fourth set is the sandbox: a research and development environment that needs be in place to run through configurations before deployment.

The internet is littered with stories of network admins who botched an update by forgetting a single character in a config file. If you are charged with running a mission-critical network, you really don't want that to be you.

This is complicated further if, like most sysadmins these days, you are not always on site when upgrades are taking place. Toying with network gear is uniquely risky because the network is your access point for remote administration.

Having a back door is important. It doesn't have to be a sexy one, just a functional one. If you run a massive hyperscale data centre, then you probably have multiple redundant fibre-optic links providing you with all sorts of ways into the data centre. Smaller organisations don't have this luxury.

Fax of life

The right equipment can help you here, and it is available dirt cheap or even free. Consider a branch office scenario. If the branch office has a fax line, you are in business.

A simple appliance called the Stick can take that fax line you are already paying for and enable you to dial in to an onsite bridgehead system in case of an emergency. Sure, dial-up is largely worthless, but it will pass an RDP session and it will reload that config file you screwed up.

It might seem anachronistic to go from talking about four layers worth of network redundancy to a dial-up modem as your last line of defence, but it is all part of the same philosophy.

If you cut this cable, how much trouble are you in?

Create a detailed network map including every device and link on your network. Consider one by one what happens if that device fails. If you cut this cable, how much trouble are you in? If you take a gun and shoot that switch, can your network survive?

The hard part is not the technology. It exists, it is proven and there are whitepapers for nearly any conceivable scenario. The hard part is the people.

The challenge is convincing the people with the money to invest in redundancy. They need to know what a network outage can mean to the bottom line. The hard part is changing a culture of "good enough" into one that understands the false economy of ignoring redundancy.

So go ahead and unplug that network cable. There's a backup...right? ®

Remote control for virtualized desktops

More from The Register

next story
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
NASA launches new climate model at SC14
75 days of supercomputing later ...
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
BOFH: WHERE did this 'fax-enabled' printer UPGRADE come from?
Don't worry about that cable, it's part of the config
Stop the IoT revolution! We need to figure out packet sizes first
Researchers test 802.15.4 and find we know nuh-think! about large scale sensor network ops
DEATH by COMMENTS: WordPress XSS vuln is BIGGEST for YEARS
Trio of XSS turns attackers into admins
SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016
Flash WORM has a serious use for archived photos and videos
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
Go beyond APM with real-time IT operations analytics
How IT operations teams can harness the wealth of wire data already flowing through their environment for real-time operational intelligence.
Why CIOs should rethink endpoint data protection in the age of mobility
Assessing trends in data protection, specifically with respect to mobile devices, BYOD, and remote employees.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?