Feeds

The Navitaire cock-up, filleted

Datacentre details behind Virgin Blue outages

Application security programs and practises

The Register has found out more about the Navitaire data centre responsible for the 21-hour Virgin Blue outage on Sunday and the second outage on Tuesday. It makes the prolonged first outage even more puzzling.

Accenture's Eric Ulmer says (Google Docs) that the Navitaire system has a high transaction volume, and the total value of web-based travel transactions exceeds Amazon.com transactions. The daily credit authorisations reach $135m and some 750,000 batch jobs are run each month.

There are high service level agreement penalties for outages, and application response time waits. There are no scheduled maintenance windows and Navitaire in effect pays to take airline customers down. The first 21-hour outage caused more than 100 Virgin Blue flights to be cancelled with more than 100,000 passengers affected.

The net effect of the two outages could be that Navitaire pays millions of Australian dollars in compensation to Virgin Blue and, perhaps, loses the airline reservation system contract.

We are told by an anonymous source that Navitaire's Sydney-based data centre has more than 32 units of Texas Memory Systems' RamSan 500 front ended by NetApp V-Series V6040 6080 controllers. Ulmer says the RamSans deliver 8TB of usable storage.

We understand Navitaire takes snapshots of the core OLTP database and SnapVaults it to nearline storage. This then gets dumped to local DataDomain systems, which Ulmer says are used as virtual tape libraries. Navitaire also SnapMirrors the primary array OLTP data to another data centre.

We already know the RamSans have a lot of internal data protection features and that Navitaire staff attempted to repair the failed "solid state disk infrastructure" as Virgin Blue described it in a statement.

The fact of these SnapVault and SnapMirror arrangements being in place raises the question why did it take 21 hours to restore the system? Couldn't Navitaire just recover the snapshot data to repaired hardware?

Alternatively, why didn't it run the system off the remote data centre holding the SnapMirror data? The implication is that, because it didn't do this, the Snapmirror process of sending OLTP data to the second data centre was useless.

Another point: It's understood the Sydney data centre services several airlines so why was only Virgin Blue affected?

It is becoming more and more obvious that Navitaire's business continuance and disaster recovery provisions failed completely in this outage. There should have been standby systems ready to take on the load of any failed system or system component, but there weren't any. That is a blunder of the first magnitude by whoever designed, implemented and ran the system. ®

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
Attack of the clones: Oracle's latest Red Hat Linux lookalike arrives
Oracle's Linux boss says Larry's Linux isn't just for Oracle apps anymore
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
Apple fanbois SCREAM as update BRICKS their Macbook Airs
Ragegasm spills over as firmware upgrade kills machines
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
prev story

Whitepapers

Top three mobile application threats
Prevent sensitive data leakage over insecure channels or stolen mobile devices.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.