Feeds

The Navitaire cock-up, filleted

Datacentre details behind Virgin Blue outages

Intelligent flash storage arrays

The Register has found out more about the Navitaire data centre responsible for the 21-hour Virgin Blue outage on Sunday and the second outage on Tuesday. It makes the prolonged first outage even more puzzling.

Accenture's Eric Ulmer says (Google Docs) that the Navitaire system has a high transaction volume, and the total value of web-based travel transactions exceeds Amazon.com transactions. The daily credit authorisations reach $135m and some 750,000 batch jobs are run each month.

There are high service level agreement penalties for outages, and application response time waits. There are no scheduled maintenance windows and Navitaire in effect pays to take airline customers down. The first 21-hour outage caused more than 100 Virgin Blue flights to be cancelled with more than 100,000 passengers affected.

The net effect of the two outages could be that Navitaire pays millions of Australian dollars in compensation to Virgin Blue and, perhaps, loses the airline reservation system contract.

We are told by an anonymous source that Navitaire's Sydney-based data centre has more than 32 units of Texas Memory Systems' RamSan 500 front ended by NetApp V-Series V6040 6080 controllers. Ulmer says the RamSans deliver 8TB of usable storage.

We understand Navitaire takes snapshots of the core OLTP database and SnapVaults it to nearline storage. This then gets dumped to local DataDomain systems, which Ulmer says are used as virtual tape libraries. Navitaire also SnapMirrors the primary array OLTP data to another data centre.

We already know the RamSans have a lot of internal data protection features and that Navitaire staff attempted to repair the failed "solid state disk infrastructure" as Virgin Blue described it in a statement.

The fact of these SnapVault and SnapMirror arrangements being in place raises the question why did it take 21 hours to restore the system? Couldn't Navitaire just recover the snapshot data to repaired hardware?

Alternatively, why didn't it run the system off the remote data centre holding the SnapMirror data? The implication is that, because it didn't do this, the Snapmirror process of sending OLTP data to the second data centre was useless.

Another point: It's understood the Sydney data centre services several airlines so why was only Virgin Blue affected?

It is becoming more and more obvious that Navitaire's business continuance and disaster recovery provisions failed completely in this outage. There should have been standby systems ready to take on the load of any failed system or system component, but there weren't any. That is a blunder of the first magnitude by whoever designed, implemented and ran the system. ®

Beginner's guide to SSL certificates

More from The Register

next story
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
BOFH: WHERE did this 'fax-enabled' printer UPGRADE come from?
Don't worry about that cable, it's part of the config
Stop the IoT revolution! We need to figure out packet sizes first
Researchers test 802.15.4 and find we know nuh-think! about large scale sensor network ops
Turnbull should spare us all airline-magazine-grade cloud hype
Box-hugger is not a dirty word, Minister. Box-huggers make the cloud WORK
SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016
Flash WORM has a serious use for archived photos and videos
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
prev story

Whitepapers

Go beyond APM with real-time IT operations analytics
How IT operations teams can harness the wealth of wire data already flowing through their environment for real-time operational intelligence.
5 critical considerations for enterprise cloud backup
Key considerations when evaluating cloud backup solutions to ensure adequate protection security and availability of enterprise data.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
How to simplify SSL certificate management
Simple steps to take control of SSL certificates across the enterprise, and recommendations centralizing certificate management throughout their lifecycle.