Feeds

The Navitaire cock-up, filleted

Datacentre details behind Virgin Blue outages

Internet Security Threat Report 2014

The Register has found out more about the Navitaire data centre responsible for the 21-hour Virgin Blue outage on Sunday and the second outage on Tuesday. It makes the prolonged first outage even more puzzling.

Accenture's Eric Ulmer says (Google Docs) that the Navitaire system has a high transaction volume, and the total value of web-based travel transactions exceeds Amazon.com transactions. The daily credit authorisations reach $135m and some 750,000 batch jobs are run each month.

There are high service level agreement penalties for outages, and application response time waits. There are no scheduled maintenance windows and Navitaire in effect pays to take airline customers down. The first 21-hour outage caused more than 100 Virgin Blue flights to be cancelled with more than 100,000 passengers affected.

The net effect of the two outages could be that Navitaire pays millions of Australian dollars in compensation to Virgin Blue and, perhaps, loses the airline reservation system contract.

We are told by an anonymous source that Navitaire's Sydney-based data centre has more than 32 units of Texas Memory Systems' RamSan 500 front ended by NetApp V-Series V6040 6080 controllers. Ulmer says the RamSans deliver 8TB of usable storage.

We understand Navitaire takes snapshots of the core OLTP database and SnapVaults it to nearline storage. This then gets dumped to local DataDomain systems, which Ulmer says are used as virtual tape libraries. Navitaire also SnapMirrors the primary array OLTP data to another data centre.

We already know the RamSans have a lot of internal data protection features and that Navitaire staff attempted to repair the failed "solid state disk infrastructure" as Virgin Blue described it in a statement.

The fact of these SnapVault and SnapMirror arrangements being in place raises the question why did it take 21 hours to restore the system? Couldn't Navitaire just recover the snapshot data to repaired hardware?

Alternatively, why didn't it run the system off the remote data centre holding the SnapMirror data? The implication is that, because it didn't do this, the Snapmirror process of sending OLTP data to the second data centre was useless.

Another point: It's understood the Sydney data centre services several airlines so why was only Virgin Blue affected?

It is becoming more and more obvious that Navitaire's business continuance and disaster recovery provisions failed completely in this outage. There should have been standby systems ready to take on the load of any failed system or system component, but there weren't any. That is a blunder of the first magnitude by whoever designed, implemented and ran the system. ®

Beginner's guide to SSL certificates

More from The Register

next story
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
Stop the IoT revolution! We need to figure out packet sizes first
Researchers test 802.15.4 and find we know nuh-think! about large scale sensor network ops
Turnbull should spare us all airline-magazine-grade cloud hype
Box-hugger is not a dirty word, Minister. Box-huggers make the cloud WORK
SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016
Flash WORM has a serious use for archived photos and videos
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
Do you spend ages wasting time because of a bulging rack?
No more cloud-latency tea breaks for you, users! Get a load of THIS
prev story

Whitepapers

Driving business with continuous operational intelligence
Introducing an innovative approach offered by ExtraHop for producing continuous operational intelligence.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
How to determine if cloud backup is right for your servers
Two key factors, technical feasibility and TCO economics, that backup and IT operations managers should consider when assessing cloud backup.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Internet Security Threat Report 2014
An overview and analysis of the year in global threat activity: identify, analyze, and provide commentary on emerging trends in the dynamic threat landscape.