Feeds

Titsup EMC VNX kit unleashes 5 days of chaos in Sweden

Crashed Tieto system took down bank, pharmacy, schools' website

Intelligent flash storage arrays

Tieto, a prominent Swedish IT service supplier, had an EMC Array go titsup on 25 November, causing five days of chaos at the Motor Vehicle Inspectorate, the Sollentuna and Nacka municipalities, the City of Stockholm's schools' website and intranet, the National Board of Health and other prominent sites.

The debacle (in Swedish) started when a VNX array at Tieto apparently went bonkers and crashed. The cache failed, the original data on disk was corrupted and the copy of the data was also corrupt.

According to the Computer Sweden (CS) media outlet (Google Translate) a Legato Networker backup of virtual machine data to tape was involved and the backup data could not be read. There is talk of the EMC (Legato) Networker client software not being compatible with Windows 2008 R2 which Tieto was using. Tieto could not read Networker tape backups on virtual Windows servers.

Swedish media reports that an upgrade at the VNX installation, to an NS480 partly for its caching, failed. This is odd as the NS480 is a Celerra product, a precursor to the VNX line which unified Clariion block and Celerra file storage in one product, albeit with two separate component operating systems. On the face of it adding an NS480 to a VNX would be a downgrade and not an upgrade.

Angry Swedes couldn't register car ownership, or have the equivalent of MOTs (vehicle inspection) carried out, medical prescriptions couldn't be processed, and many other IT services using Tieto facilities were paralysed. These included the Central Student Grants Committee, Malmo University and National Board of Health.

EMC's local headman, Robert Ekström, wouldn't talk about it to the press. Neither would Tieto's VP for IT operations, Michael Jupiter.

A CS report quoted Bo Andersson, the CIO of SBAB bank which was heavily affected, despite having a 99.8 per cent uptime agreement with Tieto: "You have to understand the magnitude of what happened. An hour's interruption is very serious, and after four hours is an emergency plan set. One hundred hours is so far beyond anything we've ever been through, I find no words ... We are deeply shocked"

An EMC spokesperson said: "We cannot comment on the specifics of any of our customers, but we are happy to share with you Tieto’s own statement which was posted to their Swedish website in December." Here is an edited version of that statement:

On Friday 25 November, an EMC storage system in one of Tieto’s data centres in Sweden experienced a rare combination of component errors. EMC responded immediately, and within 48 hours the initial storage related issue was resolved. However, the incident triggered a sequence of events requiring a complex and time consuming recovery process affecting approximately 50 of Tieto’s customers.

... Due to the complexity of the recovery process the full system recovery for some of our customers took longer than originally anticipated. Furthermore services critical to the community have been prioritized throughout the recovery process. Of the total amount of affected services, all main services have already been recovered and brought completely back online.

... Tieto and EMC are jointly conducting a technical analysis to unveil the root causes of this incident. The technical analysis will be part of a more comprehensive investigation led by an external investigator and carried out in collaboration with Swedish Civil Contingencies Agency among others. We deeply regret the inconveniences this event has caused for our customers and our customers’ customers.

An EMC array crash causing five days of disruption and data loss doesn't say much for product reliability or Tieto's business continuance arrangements, even if it is a rare occurrence. Two days before the incident EMC published a document on disaster recovery – a tad ironic with hindsight.

No doubt the root cause will be found and shared with Tieto's customers in contract negotiations under conditions of secrecy, and to satisfy them that it won't happen again. It's doubtful if the world at large will find out what happened though. ®

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
Ex-US Navy fighter pilot MIT prof: Drones beat humans - I should know
'Missy' Cummings on UAVs, smartcars and dying from boredom
Facebook, Apple: LADIES! Why not FREEZE your EGGS? It's on the company!
No biological clockwatching when you work in Silicon Valley
The 'fun-nification' of computer education – good idea?
Compulsory code schools, luvvies love it, but what about Maths and Physics?
Doctor Who's Flatline: Cool monsters, yes, but utterly limp subplots
We know what the Doctor does, stop going on about it already
'Cowardly, venomous trolls' threatened with TWO-YEAR sentences for menacing posts
UK government: 'Taking a stand against a baying cyber-mob'
Happiness economics is bollocks. Oh, UK.gov just adopted it? Er ...
Opportunity doesn't knock; it costs us instead
Sysadmin with EBOLA? Gartner's issued advice to debug your biz
Start hoarding cleaning supplies, analyst firm says, and assume your team will scatter
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.