Feeds

Titsup EMC VNX kit unleashes 5 days of chaos in Sweden

Crashed Tieto system took down bank, pharmacy, schools' website

The Essential Guide to IT Transformation

Tieto, a prominent Swedish IT service supplier, had an EMC Array go titsup on 25 November, causing five days of chaos at the Motor Vehicle Inspectorate, the Sollentuna and Nacka municipalities, the City of Stockholm's schools' website and intranet, the National Board of Health and other prominent sites.

The debacle (in Swedish) started when a VNX array at Tieto apparently went bonkers and crashed. The cache failed, the original data on disk was corrupted and the copy of the data was also corrupt.

According to the Computer Sweden (CS) media outlet (Google Translate) a Legato Networker backup of virtual machine data to tape was involved and the backup data could not be read. There is talk of the EMC (Legato) Networker client software not being compatible with Windows 2008 R2 which Tieto was using. Tieto could not read Networker tape backups on virtual Windows servers.

Swedish media reports that an upgrade at the VNX installation, to an NS480 partly for its caching, failed. This is odd as the NS480 is a Celerra product, a precursor to the VNX line which unified Clariion block and Celerra file storage in one product, albeit with two separate component operating systems. On the face of it adding an NS480 to a VNX would be a downgrade and not an upgrade.

Angry Swedes couldn't register car ownership, or have the equivalent of MOTs (vehicle inspection) carried out, medical prescriptions couldn't be processed, and many other IT services using Tieto facilities were paralysed. These included the Central Student Grants Committee, Malmo University and National Board of Health.

EMC's local headman, Robert Ekström, wouldn't talk about it to the press. Neither would Tieto's VP for IT operations, Michael Jupiter.

A CS report quoted Bo Andersson, the CIO of SBAB bank which was heavily affected, despite having a 99.8 per cent uptime agreement with Tieto: "You have to understand the magnitude of what happened. An hour's interruption is very serious, and after four hours is an emergency plan set. One hundred hours is so far beyond anything we've ever been through, I find no words ... We are deeply shocked"

An EMC spokesperson said: "We cannot comment on the specifics of any of our customers, but we are happy to share with you Tieto’s own statement which was posted to their Swedish website in December." Here is an edited version of that statement:

On Friday 25 November, an EMC storage system in one of Tieto’s data centres in Sweden experienced a rare combination of component errors. EMC responded immediately, and within 48 hours the initial storage related issue was resolved. However, the incident triggered a sequence of events requiring a complex and time consuming recovery process affecting approximately 50 of Tieto’s customers.

... Due to the complexity of the recovery process the full system recovery for some of our customers took longer than originally anticipated. Furthermore services critical to the community have been prioritized throughout the recovery process. Of the total amount of affected services, all main services have already been recovered and brought completely back online.

... Tieto and EMC are jointly conducting a technical analysis to unveil the root causes of this incident. The technical analysis will be part of a more comprehensive investigation led by an external investigator and carried out in collaboration with Swedish Civil Contingencies Agency among others. We deeply regret the inconveniences this event has caused for our customers and our customers’ customers.

An EMC array crash causing five days of disruption and data loss doesn't say much for product reliability or Tieto's business continuance arrangements, even if it is a rare occurrence. Two days before the incident EMC published a document on disaster recovery – a tad ironic with hindsight.

No doubt the root cause will be found and shared with Tieto's customers in contract negotiations under conditions of secrecy, and to satisfy them that it won't happen again. It's doubtful if the world at large will find out what happened though. ®

Build a business case: developing custom apps

More from The Register

next story
iPad? More like iFAD: We reveal why Apple fell into IBM's arms
But never fear fanbois, you're still lapping up iPhones, Macs
Sonos AXES support for Apple's iOS4 and 5
Want to use your iThing? You can't - it's too old
Amazon says Hachette should lower ebook prices, pay authors more
Oh yeah ... and a 30% cut for Amazon to seal the deal
Philip K Dick 'Nazi alternate reality' story to be made into TV series
Amazon Studios, Ridley Scott firm to produce The Man in the High Castle
Joe Average isn't worth $10 a year to Mark Zuckerberg
The Social Network deflates the PC resurgence with mobile-only usage prediction
Chips are down at Broadcom: Thousands of workers laid off
Cellphone baseband device biz shuttered
Feel free to BONK on the TUBE, says Transport for London
Plus: Almost NOBODY uses pay-by-bonk on buses - Visa
Twitch rich as Google flicks $1bn hitch switch, claims snitch
Gameplay streaming biz and search king refuse to deny fresh gobble rumors
Stick a 4K in them: Super high-res TVs are DONE
4,000 pixels is niche now... Don't say we didn't warn you
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.
Maximize storage efficiency across the enterprise
The HP StoreOnce backup solution offers highly flexible, centrally managed, and highly efficient data protection for any enterprise.