Feeds

The Navitaire cock-up, filleted

Datacentre details behind Virgin Blue outages

Intelligent flash storage arrays

The Register has found out more about the Navitaire data centre responsible for the 21-hour Virgin Blue outage on Sunday and the second outage on Tuesday. It makes the prolonged first outage even more puzzling.

Accenture's Eric Ulmer says (Google Docs) that the Navitaire system has a high transaction volume, and the total value of web-based travel transactions exceeds Amazon.com transactions. The daily credit authorisations reach $135m and some 750,000 batch jobs are run each month.

There are high service level agreement penalties for outages, and application response time waits. There are no scheduled maintenance windows and Navitaire in effect pays to take airline customers down. The first 21-hour outage caused more than 100 Virgin Blue flights to be cancelled with more than 100,000 passengers affected.

The net effect of the two outages could be that Navitaire pays millions of Australian dollars in compensation to Virgin Blue and, perhaps, loses the airline reservation system contract.

We are told by an anonymous source that Navitaire's Sydney-based data centre has more than 32 units of Texas Memory Systems' RamSan 500 front ended by NetApp V-Series V6040 6080 controllers. Ulmer says the RamSans deliver 8TB of usable storage.

We understand Navitaire takes snapshots of the core OLTP database and SnapVaults it to nearline storage. This then gets dumped to local DataDomain systems, which Ulmer says are used as virtual tape libraries. Navitaire also SnapMirrors the primary array OLTP data to another data centre.

We already know the RamSans have a lot of internal data protection features and that Navitaire staff attempted to repair the failed "solid state disk infrastructure" as Virgin Blue described it in a statement.

The fact of these SnapVault and SnapMirror arrangements being in place raises the question why did it take 21 hours to restore the system? Couldn't Navitaire just recover the snapshot data to repaired hardware?

Alternatively, why didn't it run the system off the remote data centre holding the SnapMirror data? The implication is that, because it didn't do this, the Snapmirror process of sending OLTP data to the second data centre was useless.

Another point: It's understood the Sydney data centre services several airlines so why was only Virgin Blue affected?

It is becoming more and more obvious that Navitaire's business continuance and disaster recovery provisions failed completely in this outage. There should have been standby systems ready to take on the load of any failed system or system component, but there weren't any. That is a blunder of the first magnitude by whoever designed, implemented and ran the system. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
NASA launches new climate model at SC14
75 days of supercomputing later ...
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
You think the CLOUD's insecure? It's BETTER than UK.GOV's DATA CENTRES
We don't even know where some of them ARE – Maude
DEATH by COMMENTS: WordPress XSS vuln is BIGGEST for YEARS
Trio of XSS turns attackers into admins
BOFH: WHERE did this 'fax-enabled' printer UPGRADE come from?
Don't worry about that cable, it's part of the config
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
How to determine if cloud backup is right for your servers
Two key factors, technical feasibility and TCO economics, that backup and IT operations managers should consider when assessing cloud backup.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Choosing a cloud hosting partner with confidence
Download Choosing a Cloud Hosting Provider with Confidence to learn more about cloud computing - the new opportunities and new security challenges.